From jay.krell at cornell.edu Fri Jul 1 04:34:02 2016 From: jay.krell at cornell.edu (Jay K) Date: Fri, 1 Jul 2016 02:34:02 +0000 Subject: [M3devel] slight IR variation among targets? Message-ID: This is both a question and an explanation of hopefully an uncoming chnage, if I figure it out. Most platforms are almost the same. For example I386_FREEBSD and I386_NETBSD. You want their IR (and possibly code, but here just the IR) to generally be the same, along as long they aren't printing their actual target name. And they mostly are. Here is a different for example: jair:libm3 jay$ diff -du I386_NETBSD/NullRd.mc.txt I386_FREEBSD/NullRd.mc.txt? --- I386_NETBSD/NullRd.mc.txt 2016-06-30 19:13:11.000000000 -0700 +++ I386_FREEBSD/NullRd.mc.txt 2016-06-30 19:13:23.000000000 -0700 @@ -44,7 +44,6 @@ ? declare_procedure NullRd__Length 1 Int.32 0 0 F * p.7 ? declare_param rd 4 4 Addr 34129018 F F 50 v.10 ? declare_local _result 4 4 Int.32 425470580 F F 50 v.11 - reveal_opaque 34129018 -885236436 ? declare_opaque 483796623 -1651526519 ? declare_proctype -2116580915 2 -915982019 2 0 ? declare_formal n -1746782238 @@ -85,6 +84,7 @@ ? declare_field closed 416 8 509158269 ? declare_field seekable 424 8 509158269 ? declare_field intermittent 432 8 509158269 + reveal_opaque 34129018 -885236436 ? # Init ? -----LINE 22 ?----- ? begin_procedure p.5 One line of the IR is moved. The meaning is the same. What causes this? Do we output sometimes in hash order? And target affects that? Or when buffers fill up, and target length can affect that? I'd like to remove these differences. If it is a buffer length matter, I might just expand the buffers, as systems have far far more memory than when this was all written. If it is hash ordering matter -- one should avoid outputting data in hash order. ?- Jay From jay.krell at cornell.edu Fri Jul 1 08:48:57 2016 From: jay.krell at cornell.edu (Jay K) Date: Fri, 1 Jul 2016 06:48:57 +0000 Subject: [M3devel] lang.opt with -y twice? Message-ID: any idea why -y is here twice? Seems redundant. https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt?rev=1.2;content-type=text%2Fplain -y is shorthand for trace all, I know, Tony's advise years ago, very useful ?- Jay From rodney_bates at lcwb.coop Fri Jul 1 17:01:51 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Fri, 01 Jul 2016 10:01:51 -0500 Subject: [M3devel] source paths to generics? In-Reply-To: References: , , , , , , , <57746905.2080007@lcwb.coop> , <5775407E.6000401@lcwb.coop> Message-ID: <577685DF.2050404@lcwb.coop> On 06/30/2016 11:11 AM, Jay K wrote: > I put in a bunch of calls to RTIO.PutText("I'm here 1"); RTIO.Flush(), RTIO.PutText("I'm here 2"); RTIO.Flush(). > > When I saw those I knew I missed it somehow and looked again. > > I still don't see the strings in the output though. > > - Jay > > The missing call to M3Front.ParseImports is at cm3/src/Builder.m3:2007. It's in my previous grep output, plain as day. I must have just missed it in all the noise. But I'm still pretty sure the "=>" ("1=>"?) inserted by M3Header.Parse will never find its way into any output. > > ---------------------------------------- >> Date: Thu, 30 Jun 2016 10:53:34 -0500 >> From: rodney_bates at lcwb.coop >> To: m3devel at elegosoft.com >> Subject: Re: [M3devel] source paths to generics? >> >> >> >> On 06/29/2016 11:49 PM, Jay K wrote: >>> I guess neither of us looked outside of m3front. >>> The code runs. Not clear if the strings can get output. >>> Maybe from asserts or Compiler.ThisFile in a generic? >>> I"ll try some more. >>> >> >> I started looking in all of m3-sys, since the different compiler >> packages there are linked together into cm3. Later, I >> looked in the entire cm3 repo and didn't see anything. >> In any case, since it's part of the compiler, it really would be >> strange to call it from outside m3-sys. >> >> But if it is executing, we must have missed something. How did you >> find it executing? In a debugger? Can you do a backtrace? >> >>> jair:m3core jay$ grep -i parseimp ../../m3-sys/cm3/src/Builder.m3 >>> ids := M3Front.ParseImports (source, s.m3env); >>> >>> >>> - Jay >>> >>> >>> ---------------------------------------- >>>> Date: Wed, 29 Jun 2016 19:34:13 -0500 >>>> From: rodney_bates at lcwb.coop >>>> To: m3devel at elegosoft.com >>>> Subject: Re: [M3devel] source paths to generics? >>>> >>>> Hmm. I poked around a bit, and it looks like there is some cruft here, >>>> left over from something. >>>> >>>> The only call I can find on M3Header.Parse is M3Front.m3:37. This is >>>> in M3Front.ParseImports. I can find no calls on that at all. (There >>>> are other procedures by this name.) >>>> >>>> Also, it appears M3Header.ParseImports actually collects the exports, the imports, >>>> the generic actuals. >>>> >>>> As for the concocted (by M3Header) name of the form => , >>>> this looks to be used only to initialize the scanner's file number while scanning >>>> up through the formals of the generic, but that is not used. >>>> >>>> On 06/29/2016 11:34 AM, Jay K wrote: >>>>> There is similar code in Module.m3 and that is the code producing >>>>> the data "I don't like". >>>>> >>>>> M3Header isn't dead but I haven't seen it run yet. >>>>> >>>>> I changed them from "=>" to "1=>" and "2=>" and looked >>>>> where those occur in the output .s files under -keep. >>>>> >>>>> This is kinda something I wish were easier, like more strings >>>>> need to be longer for easier finding w/o ambiguity (the flip >>>>> side of my context arguments!) >>>>> >>>>> As things stand, I can't check that in. >>>>> I suppose maybe with a CG.comment but nevermind. >>>>> >>>>> - Jay >>>>> >>>>> >>>>> >>>>> ---------------------------------------- >>>>>> From: jay.krell at cornell.edu >>>>>> To: m3devel at elegosoft.com >>>>>> Subject: RE: [M3devel] source paths to generics? >>>>>> Date: Wed, 29 Jun 2016 15:22:14 +0000 >>>>>> >>>>>> mfront/src/misc/M3Header.m3 has this: >>>>>> >>>>>> >>>>>> (* build a synthetic file name & start reading *) >>>>>> filename := old_filename & " => " & filename; >>>>>> Scanner.Push (filename, s.generic, is_main := Scanner.in_main); >>>>>> >>>>>> >>>>>> and instantiated generics aren't what I thought. >>>>>> I thought the build system or compiler did a textual >>>>>> substitution of the generic parameters into the entire implementation, >>>>>> and saved that to the file system, >>>>>> and had the assert/debug paths point to it. >>>>>> >>>>>> So could step through what looks exactly like an .m3 file. >>>>>> >>>>>> But it doesn't. The instantiated file is a little stub, like: >>>>>> >>>>>> jair:libm3 jay$ more I386_DARWIN/TextAtomTbl.m3 >>>>>> (*generated by m3build*) >>>>>> >>>>>> MODULE TextAtomTbl = Table (Text, Atom) END TextAtomTbl. >>>>>> >>>>>> >>>>>> so I have one or two proposals. >>>>>> >>>>>> 1) old_filename above contains the target, it looks like: >>>>>> >>>>>> "../I386_DARWIN/TextAtomTbl.m3 => ../src/table/Table.mg" >>>>>> or >>>>>> "../I386_DARWIN/TextAtomTbl.m3 => ../src/table" >>>>>> >>>>>> These both occur. >>>>>> I'm not sure what the second is, seems like a mistake. >>>>>> >>>>>> I suggest the first string in both pairs be changed to be the leaf element, like: >>>>>> >>>>>> "TextAtomTbl.m3 => ../src/table/Table.mg" >>>>>> or >>>>>> "TextAtomTbl.m3 => ../src/table" >>>>>> >>>>>> and maybe the second string in both cases should be as it is in the second pair. >>>>>> >>>>>> 2) Possibly it should reall just be: >>>>>> ../src/table/Table.mg >>>>>> >>>>>> and developer can step through that, knowing that it is a somewhat abstracted >>>>>> form of what is running >>>>>> >>>>>> I'm willing to do this under like: >>>>>> >>>>>> CONST TemporaryForTargetConvergence = TRUE >>>>>> >>>>>> or >>>>>> VAR TemporaryForTargetConvergence: BOOLEAN; >>>>>> >>>>>> and a command line switch, but I suspect it isn't really useful to anyone asis, >>>>>> so might as well remove target unconditionally. >>>>>> >>>>>> >>>>>> ? >>>>>> >>>>>> - Jay >>>>>> >>>>>> >>>>>> >>>>>> ---------------------------------------- >>>>>>> From: jay.krell at cornell.edu >>>>>>> To: m3devel at elegosoft.com >>>>>>> Date: Wed, 29 Jun 2016 09:44:51 +0000 >>>>>>> Subject: [M3devel] source paths to generics? >>>>>>> >>>>>>> I haven't fully verified this, but it appears for example if I debug a generic, or fail an assert in a generic, >>>>>>> the source file I am told about is the instantiated i3/m3 file. >>>>>>> >>>>>>> This isn't particularly useful for the programmer or convenient for the compiler, right? >>>>>>> >>>>>>> The programmer would rather see the .ig/.mg files, and this is easy to provide in the compiler? >>>>>>> I guess it is slightly easier in the compiler, but easy to do better? >>>>>>> >>>>>>> I should look into make it so? >>>>>>> >>>>>>> My real agenda is I want to see: >>>>>>> ../src/foo.mg >>>>>>> >>>>>>> instead of >>>>>>> I386_DARWIN/foo.m3 >>>>>>> >>>>>>> I don't want the target variation, but other points seem true also, right? >>>>>>> >>>>>>> Right? The line numbers match between them, and the generic syntax is so close to "normal" that a programmer not used to it won't be confused, right? >>>>>>> >>>>>>> I'll try to poke around more. >>>>>>> >>>>>>> - Jay >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> M3devel mailing list >>>>>>> M3devel at elegosoft.com >>>>>>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>>>>> >>>>> >>>>> _______________________________________________ >>>>> M3devel mailing list >>>>> M3devel at elegosoft.com >>>>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>>>> >>>> >>>> -- >>>> Rodney Bates >>>> rodney.m.bates at acm.org >>>> _______________________________________________ >>>> M3devel mailing list >>>> M3devel at elegosoft.com >>>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>> >>> _______________________________________________ >>> M3devel mailing list >>> M3devel at elegosoft.com >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>> >> >> -- >> Rodney Bates >> rodney.m.bates at acm.org >> _______________________________________________ >> M3devel mailing list >> M3devel at elegosoft.com >> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > -- Rodney Bates rodney.m.bates at acm.org From wagner at elegosoft.com Fri Jul 1 17:17:10 2016 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 1 Jul 2016 17:17:10 +0200 Subject: [M3devel] source paths to generics? In-Reply-To: <577685DF.2050404@lcwb.coop> References: <57746905.2080007@lcwb.coop> <5775407E.6000401@lcwb.coop> <577685DF.2050404@lcwb.coop> Message-ID: <20160701171710.add85dcbf03e0e3f1f24c451@elegosoft.com> On Fri, 01 Jul 2016 10:01:51 -0500 "Rodney M. Bates" wrote: > On 06/30/2016 11:11 AM, Jay K wrote: > > I put in a bunch of calls to RTIO.PutText("I'm here 1"); RTIO.Flush(), RTIO.PutText("I'm here 2"); RTIO.Flush(). > > > > When I saw those I knew I missed it somehow and looked again. > > > > I still don't see the strings in the output though. > > > > - Jay > > > > > > The missing call to M3Front.ParseImports is at cm3/src/Builder.m3:2007. > It's in my previous grep output, plain as day. > I must have just missed it in all the noise. > > But I'm still pretty sure the "=>" ("1=>"?) inserted by M3Header.Parse will > never find its way into any output. I'm sure there's lots of old and obsolete code in the packages, as CM3 was built upon the DEC SRC compiler sources. The principal programmer of the compiler and Reactor is easily found via LinkedIn: https://www.linkedin.com/in/billkalsow I'm not sure if he is willing to answer questions about his work, but if you run into a real problem, it might be worthwhile to try it. Olaf -- Olaf Wagner -- elego Software Solutions GmbH -- http://www.elegosoft.com Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From rodney_bates at lcwb.coop Fri Jul 1 20:24:48 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Fri, 01 Jul 2016 13:24:48 -0500 Subject: [M3devel] source paths to generics? In-Reply-To: <20160701171710.add85dcbf03e0e3f1f24c451@elegosoft.com> References: <57746905.2080007@lcwb.coop> <5775407E.6000401@lcwb.coop> <577685DF.2050404@lcwb.coop> <20160701171710.add85dcbf03e0e3f1f24c451@elegosoft.com> Message-ID: <5776B570.1050504@lcwb.coop> On 07/01/2016 10:17 AM, Olaf Wagner wrote: > On Fri, 01 Jul 2016 10:01:51 -0500 > "Rodney M. Bates" wrote: > >> On 06/30/2016 11:11 AM, Jay K wrote: >>> I put in a bunch of calls to RTIO.PutText("I'm here 1"); RTIO.Flush(), RTIO.PutText("I'm here 2"); RTIO.Flush(). >>> >>> When I saw those I knew I missed it somehow and looked again. >>> >>> I still don't see the strings in the output though. >>> >>> - Jay >>> >>> >> >> The missing call to M3Front.ParseImports is at cm3/src/Builder.m3:2007. >> It's in my previous grep output, plain as day. >> I must have just missed it in all the noise. >> >> But I'm still pretty sure the "=>" ("1=>"?) inserted by M3Header.Parse will >> never find its way into any output. > > I'm sure there's lots of old and obsolete code in the packages, as CM3 > was built upon the DEC SRC compiler sources. > I didn't make this clear, but, with the newly discovered call, my disparaging remarks about M3Header.Parse, etc. are now refuted. > The principal programmer of the compiler and Reactor is easily found > via LinkedIn: https://www.linkedin.com/in/billkalsow > > I'm not sure if he is willing to answer questions about his work, but > if you run into a real problem, it might be worthwhile to try it. > > Olaf > -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Fri Jul 1 20:43:20 2016 From: jay.krell at cornell.edu (Jay K) Date: Fri, 1 Jul 2016 18:43:20 +0000 Subject: [M3devel] source paths to generics? In-Reply-To: <5776B570.1050504@lcwb.coop> References: <57746905.2080007@lcwb.coop> <5775407E.6000401@lcwb.coop> <577685DF.2050404@lcwb.coop>, <20160701171710.add85dcbf03e0e3f1f24c451@elegosoft.com>, <5776B570.1050504@lcwb.coop> Message-ID: >> But I'm still pretty sure the "=>" ("1=>"?) inserted by M3Header.Parse will>> never find its way into any output. agreed.Anyway, I'm almost done with my minor change here. I'm leaning toward: cm3 -trim-target-variation will set Target.TrimTargetVariation will also a quake variable CM3_TRIM_TARGET_VARIATION maybe quake function backend() will use quake variable to pass it on to cm3cg probably Unless maybe builder code knows it is calling cm3cg and will it in options there i.e. don't intend to push this through llvm backends at this time. Used by M3C to turn off line directives entirely (vs. just commenting them out as it is willing to do) Used by M3C to turn off a comment as what is TARGET cm3cg -ftrim-target-variation used by dbxout.c and dwarf2out.c to omit current working directory It is a funny name and a funny behavior and of limited use. The exercise is somewhat useful to see how to plumb options through. Much target variation occurs, just that slightly unnecessary variation will not, and then targets that are nearly the same, will be closer to the same. In particular, it likely that there are approximately one-two ABI per architecture on Posix hosts, therefore the Posix targets are all the same -- NetBSD, OpenBSD, Solaris, Darwin, Linux, FreeBSD. Darwin will tend to vary because it assumes newer processors so favors SSE over x87. But still for AMD64, probably the same. Windows tends to have different ABIs so the IR is still the same, but the assembly is not. - Jay > Date: Fri, 1 Jul 2016 13:24:48 -0500 > From: rodney_bates at lcwb.coop > To: wagner at elegosoft.com; jay.krell at cornell.edu > Subject: Re: [M3devel] source paths to generics? > CC: m3devel at elegosoft.com > > > > On 07/01/2016 10:17 AM, Olaf Wagner wrote: > > On Fri, 01 Jul 2016 10:01:51 -0500 > > "Rodney M. Bates" wrote: > > > >> On 06/30/2016 11:11 AM, Jay K wrote: > >>> I put in a bunch of calls to RTIO.PutText("I'm here 1"); RTIO.Flush(), RTIO.PutText("I'm here 2"); RTIO.Flush(). > >>> > >>> When I saw those I knew I missed it somehow and looked again. > >>> > >>> I still don't see the strings in the output though. > >>> > >>> - Jay > >>> > >>> > >> > >> The missing call to M3Front.ParseImports is at cm3/src/Builder.m3:2007. > >> It's in my previous grep output, plain as day. > >> I must have just missed it in all the noise. > >> > >> But I'm still pretty sure the "=>" ("1=>"?) inserted by M3Header.Parse will > >> never find its way into any output. > > > > I'm sure there's lots of old and obsolete code in the packages, as CM3 > > was built upon the DEC SRC compiler sources. > > > > I didn't make this clear, but, with the newly discovered call, my > disparaging remarks about M3Header.Parse, etc. are now refuted. > > > The principal programmer of the compiler and Reactor is easily found > > via LinkedIn: https://www.linkedin.com/in/billkalsow > > > > I'm not sure if he is willing to answer questions about his work, but > > if you run into a real problem, it might be worthwhile to try it. > > > > Olaf > > > > -- > Rodney Bates > rodney.m.bates at acm.org > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Jul 4 10:45:23 2016 From: jay.krell at cornell.edu (Jay K) Date: Mon, 4 Jul 2016 08:45:23 +0000 Subject: [M3devel] defaulting to parallel backend? Message-ID: I suggest we make parallel backend the default. You can turn it off with -pb 1. Turn it "up" with -pb And the default is a thread per processor. This will be kernel specific code, but we can get far with ?1) Win32 GetSystemInfo, if applicable (e.g. C backend) ?2)?sysconf(_SC_NPROCESSORS_ONLN) and I'll research the various OSes (Darwin, Linux, *BSD, Solaris, and possibly AIX, HP-UX, Irix, Mingw, Cygwin). ok? I could really use the speed boost and multi-processor machines are overwhelmingly common now (though easily avoided with VMs). ?- Jay From jay.krell at cornell.edu Mon Jul 4 10:53:44 2016 From: jay.krell at cornell.edu (Jay K) Date: Mon, 4 Jul 2016 08:53:44 +0000 Subject: [M3devel] cm3 -reduce-target-variation / cm3cg -freduce-target-variation? Message-ID: I guess I should learn to use branches and/or pull requests. This adds cm3 -reduce-target-variation and cm3cg -freduce-target-variation. This meant for temporary development purposes. No user would use them. The point is to make e..g I386_FREEBSD and I386_NETBSD demonstrably identical, or at least closer to it. Ok to commit? Maybe name the two switches the same? Or something else? Maybe be less aggressive? ?i.e. even ../src/foo.m3 is getting changed to foo.m3. ?The real target is ../I386_DARWIN/foo.m3 => foo.m3? ? ? diff --git a/m3-sys/cm3/src/Main.m3 b/m3-sys/cm3/src/Main.m3 index 5d753e6..61b3f18 100644 --- a/m3-sys/cm3/src/Main.m3 +++ b/m3-sys/cm3/src/Main.m3 @@ -5,7 +5,7 @@ MODULE Main; ? ?IMPORT M3Timers, Pathname, Process, Quake; ?IMPORT RTCollector, RTParams, RTutils, Thread, Wr; -IMPORT TextTextTbl; +IMPORT TextTextTbl, Target; ? ?IMPORT Builder, Dirs, M3Build, M3Options, Makefile, Msg, Utils, WebFile; ?IMPORT MxConfig(*, M3Config, CMKey, CMCurrent *); @@ -18,6 +18,8 @@ VAR ? ?build_dir : TEXT ? ? ? ? ?:= NIL; ? ?mach ? ? ?: Quake.Machine := NIL; ? +CONST BoolToText = ARRAY BOOLEAN OF TEXT{"FALSE", "TRUE"}; + ?PROCEDURE DefineIfNotDefined (qmachine: Quake.Machine; ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?symbol, value: TEXT) RAISES {Quake.Error} = ?BEGIN @@ -73,6 +75,7 @@ VAR defs: TextTextTbl.T; ? ? ? ? ?(* DefineIfNotDefined (mach, "THREAD_LIBRARY", Version.ThreadLibrary); *) ? ? ? ? ?(* DefineIfNotDefined (mach, "WINDOW_LIBRARY", Version.WindowLibrary); *) ? ? ? ? ?DefineIfNotDefined (mach, "WORD_SIZE", MxConfig.HOST_WORD_SIZE); + ? ? ? ?DefineIfNotDefined (mach, "REDUCE_TARGET_VARIATION", BoolToText[Target.ReduceTargetVariation]); ? ? ? ? ? ?(* Even if the config file overrides the defaults, such as to do ? ? ? ? ? ? a cross build, the host characteristics are still available. *) diff --git a/m3-sys/cm3/src/Makefile.m3 b/m3-sys/cm3/src/Makefile.m3 index 64e169e..8b12320 100644 --- a/m3-sys/cm3/src/Makefile.m3 +++ b/m3-sys/cm3/src/Makefile.m3 @@ -5,7 +5,7 @@ MODULE Makefile; ? ?IMPORT FS, M3File, M3Timers, OSError, Params, Process, Text, Thread, Wr; ?IMPORT Arg, M3Build, M3Options, M3Path, Msg, Utils, TextSeq, TextTextTbl; -IMPORT MxConfig, Dirs, Version; +IMPORT MxConfig, Dirs, Version, Target; ? ?TYPE ? ?NK = M3Path.Kind; @@ -267,6 +267,9 @@ PROCEDURE ConvertOption (VAR s: State; ?arg: TEXT; ?arg_len: INTEGER) ? ? ?| 'r' => IF Text.Equal(arg, "-realclean") THEN ? ? ? ? ? ? ? ? ok := TRUE; ?(* mode set during the pre-scan *) ? ? ? ? ? ? ? ? s.found_work := TRUE; + ? ? ? ? ? ? ELSIF Text.Equal(arg, "-reduce-target-variation") THEN + ? ? ? ? ? ? ? ok := TRUE; + ? ? ? ? ? ? ? Target.ReduceTargetVariation := TRUE; ? ? ? ? ? ? ? END; ? ? ? ? ? ? ?? ? ? ?| 's' => IF Text.Equal (arg, "-silent") THEN @@ -707,6 +710,9 @@ CONST ? ? ?" ?-group-writable \"", ? ? ?" ?-pb ? ? ? ?allow parallelism in running back-end (experimental)", ? ? ?" ?-no-m3ship-resolution use quake variables in .M3SHIP (experimental)", + ? ?" ?-reduce-target-variation omit target in some minor places such as", + ? ?" ? ? ? ? ? ? ? ? ?current working directory in debug information", + ? ?" ? ? ? ? ? ? ? ? ?for internal development purposes (showing target equivalence)", ? ? ?"", ? ? ?"environment variables:", ? ? ?" ?M3CONFIG ? ? ? platform dependent configuration file to use (cm3.cfg)", diff --git a/m3-sys/cminstall/src/config-no-install/cm3cfg.common b/m3-sys/cminstall/src/config-no-install/cm3cfg.common index 7334252..9806c3c 100644 --- a/m3-sys/cminstall/src/config-no-install/cm3cfg.common +++ b/m3-sys/cminstall/src/config-no-install/cm3cfg.common @@ -460,6 +460,12 @@ proc GetM3Back() is ? ?end end ? ? ?m3back = "@" & m3back & "cm3cg " & GetM3BackFlags() + ? + ?if defined ("REDUCE_TARGET_VARIATION") + ? ?if REDUCE_TARGET_VARIATION + ? ? ?m3back = m3back & " -freduce-target-variation" + ? ?end + ?end ? ?return m3back ?end ? diff --git a/m3-sys/m3back/src/M3C.m3 b/m3-sys/m3back/src/M3C.m3 index f6740c7..22c6ade 100644 --- a/m3-sys/m3back/src/M3C.m3 +++ b/m3-sys/m3back/src/M3C.m3 @@ -29,7 +29,7 @@ VAR CaseDefaultAssertFalse := FALSE; ? ?(* Taken together, these help debugging, as you get more lines in the ? ? C and the error messages reference C line numbers *) - ?CONST output_line_directives = TRUE; + ?VAR ?output_line_directives := TRUE; ? ?CONST output_extra_newlines = FALSE; ? ?CONST inline_extract = FALSE; ? @@ -2197,7 +2197,9 @@ BEGIN ? ? ?END; ? ? ? ?IF (*self.suppress_line_directive < 1 AND*) text_last_char = '\n' THEN + ? ? ? ?IF NOT Target.ReduceTargetVariation THEN ? ? ? ? ? ?Wr.PutText(self.c, self.line_directive); + ? ? ? ?END; ? ? ? ? ?self.width := 0; ? ? ? ? ?self.last_char_was_newline := TRUE; ? ? ? ? ?RETURN; @@ -2205,7 +2207,9 @@ BEGIN ? ? ? ?IF Text.FindChar(text, '\n') # -1 THEN ? ? ? ? ?self.width := 0; (* roughly *) + ? ? ? ?IF NOT Target.ReduceTargetVariation THEN ? ? ? ? ? ?Wr.PutText(self.c, self.nl_line_directive); + ? ? ? ?END; ? ? ? ? ?self.last_char_was_newline := TRUE; ? ? ? ? ?RETURN; ? ? ?END; @@ -2217,11 +2221,13 @@ BEGIN ? ? ?END; ? ? ? ?self.width := 0; + ? ?IF NOT Target.ReduceTargetVariation THEN ? ? ? ?IF self.last_char_was_newline THEN ? ? ? ? ?Wr.PutText(self.c, self.line_directive); ? ? ? ?ELSE ? ? ? ? ?Wr.PutText(self.c, self.nl_line_directive); ? ? ? ?END; + ? ?END; ? ? ?self.last_char_was_newline := TRUE; ?END print; ? @@ -2339,9 +2345,10 @@ END set_error_handler; ?PROCEDURE Prefix_Print(self: T; multipass: Multipass_t) = ?BEGIN ? ? ?self.comment("begin unit"); + ? ?output_line_directives := output_line_directives AND NOT Target.ReduceTargetVariation; + ? ?IF NOT Target.ReduceTargetVariation THEN ? ? ? ?self.comment("M3_TARGET = ", Target.System_name); - ? ?(* This is an unnecessary target-specific output. *) - ? ?(* self.comment("M3_TARGET = ", Target.System_name); *) + ? ?END; ? ? ?self.comment("M3_WORDSIZE = ", IntToDec(Target.Word.size)); ? ? ?self.static_link_id := M3ID.Add("_static_link"); ? ? ?self.alloca_id := M3ID.Add("alloca"); diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c index dc52576..33f8844 100644 --- a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c +++ b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c @@ -1065,6 +1065,9 @@ dbxout_init (const char *input_file_name) ? ? ? labels. ?*/ ? ?ASM_GENERATE_INTERNAL_LABEL (ltext_label_name, "Ltext", 0); ? + ?/* Limit paths in debug output, to limit target variation. */ + ?if (!reduce_target_variation) + ?{ ? ? ?/* Put the current working directory in an N_SO symbol. ?*/ ? ? ?if (use_gnu_debug_info_extensions && !NO_DBX_MAIN_SOURCE_DIRECTORY) ? ? ?{ @@ -1087,6 +1090,7 @@ dbxout_init (const char *input_file_name) ? ? ? ?used_ltext_label_name = true; ?#endif /* no DBX_OUTPUT_MAIN_SOURCE_DIRECTORY */ ? ? ?} + ?} ? ? ?mapped_name = remap_debug_filename (input_file_name); ?#ifdef DBX_OUTPUT_MAIN_SOURCE_FILENAME diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c index 0da1021..0a48a65 100644 --- a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c +++ b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c @@ -15450,19 +15450,20 @@ add_gnat_descriptive_type_attribute (dw_die_ref die, tree type, ?static void ?add_comp_dir_attribute (dw_die_ref die) ?{ + ?/* Limit paths in debug output, to limit target variation. */ + ?if (reduce_target_variation) + ? ?return; + ? ?const char *wd = get_src_pwd (); - ?char *wd1; ? ? ?if (wd == NULL) ? ? ?return; ? ? ?if (DWARF2_DIR_SHOULD_END_WITH_SEPARATOR) ? ? ?{ - ? ? ?int wdlen; - - ? ? ?wdlen = strlen (wd); - ? ? ?wd1 = (char *) ggc_alloc_atomic (wdlen + 2); - ? ? ?strcpy (wd1, wd); + ? ? ?int const wdlen = (int)strlen (wd); + ? ? ?char * const wd1 = (char *) ggc_alloc_atomic (wdlen + 2); + ? ? ?memcpy (wd1, wd, wdlen); ? ? ? ?wd1 [wdlen] = DIR_SEPARATOR; ? ? ? ?wd1 [wdlen + 1] = 0; ? ? ? ?wd = wd1; diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c index 63e4b92..d468632 100644 --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c @@ -217,11 +217,8 @@ const char * ?get_src_pwd (void) ?{ ? ?if (! src_pwd) - ? ?{ - ? ? ?src_pwd = getpwd (); - ? ? ?if (!src_pwd) + ? ?if (reduce_target_variation || !(src_pwd = getpwd ())) ? ? ? ?src_pwd = "."; - ? ?} ? ? ? return src_pwd; ?} diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h index 588cfdb..f4f7cc7 100644 --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h @@ -80,4 +80,6 @@ extern bool set_src_pwd ? ? ? (const char *); ?extern HOST_WIDE_INT get_random_seed (bool); ?extern const char *set_random_seed (const char *); ? +extern bool reduce_target_variation; + ?#endif /* ! GCC_TOPLEV_H */ diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt index 3bd0469..edfca96 100644 --- a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt +++ b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt @@ -28,9 +28,6 @@ m3cg ?Language ?M3CG ? -y -m3cg M3CG - ?fopcodes-trace ?m3cg M3CG ?Trace opcodes @@ -59,10 +56,17 @@ ftypes-trace ?m3cg M3CG ?Trace types ? +freduce-target-variation +m3cg M3CG +Reduce target variation somewhat, such as by omitting current working +directory from debug info. Many necessary target variations remain. + ?v ?m3cg M3CG +print version ? ?y ?m3cg M3CG +Trace opcodes ? ?; This comment is to ensure we retain the blank line above. diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c index 03417ed..0cfa46a 100644 --- a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c +++ b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c @@ -246,6 +246,7 @@ build_case_label (tree low_value, tree high_value, tree label_decl) ?/*======================================================= OPTION HANDLING ===*/ ? ?static int option_trace_all; +bool reduce_target_variation; ? ?/*===========================================================================*/ ? @@ -6364,6 +6365,10 @@ m3_handle_option (size_t code, PCSTR /*arg*/, int /*value*/) ? ? ?case OPT_ftypes_trace: ? ? ? ?option_trace_all += 1; ? ? ? ?break; + ? ? ? + ? ?case OPT_freduce_target_variation: + ? ? ?reduce_target_variation = true; + ? ? ?break; ? ? ?} ? ? ?return 1; diff --git a/m3-sys/m3front/src/misc/Coverage.m3 b/m3-sys/m3front/src/misc/Coverage.m3 index c04c902..73fff21 100644 --- a/m3-sys/m3front/src/misc/Coverage.m3 +++ b/m3-sys/m3front/src/misc/Coverage.m3 @@ -77,8 +77,9 @@ PROCEDURE NoteProcedure (v: Value.T) = ?PROCEDURE GenerateTables () = ? ?VAR ? ? ?nLines ? ?:= MAX (0, maxLine - minLine) + 1; + ? ?fname ? ? := Host.FileTail (Host.filename); ? ? ?l_header ?:= TLen (Header); - ? ?l_fname ? := TLen (Host.filename); + ? ?l_fname ? := TLen (fname); ? ? ?l_trailer := TLen (Trailer); ? ? ?size ? ?: INTEGER; ? ? ?p ? ? ? : ProcHead; @@ -124,10 +125,10 @@ PROCEDURE GenerateTables () = ? ? ?(* CG.Init_int (size, Target.Integer.size, TInt.Zero, FALSE); *) ? ? ?INC (size, Target.Integer.size); ? ? ? ? ? ? (*timestamp*) ? - ? ?CG.Init_intt (size, Target.Integer.size, Text.Length (Host.filename), FALSE); + ? ?CG.Init_intt (size, Target.Integer.size, Text.Length (fname), FALSE); ? ? ?INC (size, Target.Integer.size); ? ? ? ? ? ? (*fileLen*) ? - ? ?CG.Init_chars (size, Host.filename, FALSE); + ? ?CG.Init_chars (size, fname, FALSE); ? ? ?INC (size, l_fname * Target.Char.size); ? ? ?(*file*) ? ? ? ?CG.Init_intt (size, Target.Integer.size, minLine, FALSE); diff --git a/m3-sys/m3front/src/misc/Host.i3 b/m3-sys/m3front/src/misc/Host.i3 index c71489f..af6a89a 100644 --- a/m3-sys/m3front/src/misc/Host.i3 +++ b/m3-sys/m3front/src/misc/Host.i3 @@ -75,4 +75,7 @@ PROCEDURE OpenUnit (name: M3ID.T; interface, generic: BOOLEAN; ? ?PROCEDURE CloseFile (rd: File.T); ? +PROCEDURE FileTail (path: TEXT): TEXT; + (* returns the 'tail' of 'path' -- after any slashes or even spaces *) + ?END Host. diff --git a/m3-sys/m3front/src/misc/Host.m3 b/m3-sys/m3front/src/misc/Host.m3 index 962d3c6..79042e3 100644 --- a/m3-sys/m3front/src/misc/Host.m3 +++ b/m3-sys/m3front/src/misc/Host.m3 @@ -9,7 +9,7 @@ ? ?MODULE Host; ? -IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler; +IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler, Target; ? ?PROCEDURE Initialize (READONLY options: ARRAY OF TEXT): BOOLEAN = ? ?BEGIN @@ -192,5 +192,24 @@ PROCEDURE CloseFile (rd: File.T) = ? ? ?END; ? ?END CloseFile; ? +PROCEDURE FileTail (path: TEXT): TEXT = + ?VAR c: CHAR; + ?BEGIN + ? ?IF NOT Target.ReduceTargetVariation THEN RETURN path; END; + + ? ?IF (path = NIL) THEN RETURN NIL END; + + ? ?(* search for the last slash or blank in the string *) + ? ?FOR x := Text.Length (path) - 1 TO 0 BY -1 DO + ? ? ?c := Text.GetChar (path, x); + ? ? ?IF (c = '/') OR (c = ' ') OR (c = '\\') THEN + ? ? ? ?RETURN Text.Sub (path, x+1); + ? ? ?END; + ? ?END; + + ? ?(* no slashes *) + ? ?RETURN path; + ?END FileTail; + ?BEGIN ?END Host. diff --git a/m3-sys/m3front/src/misc/M3Header.m3 b/m3-sys/m3front/src/misc/M3Header.m3 index 1e4decf..877d77c 100644 --- a/m3-sys/m3front/src/misc/M3Header.m3 +++ b/m3-sys/m3front/src/misc/M3Header.m3 @@ -104,7 +104,7 @@ PROCEDURE PushGeneric (VAR s: State) = ? ? ?IF (s.generic = NIL) THEN s.failed := TRUE; ?RETURN; END; ? ? ? ?(* build a synthetic file name & start reading *) - ? ?filename := old_filename & " => " & filename; + ? ?filename := Host.FileTail(old_filename) & " => " & filename; ? ? ?Scanner.Push (filename, s.generic, is_main := Scanner.in_main); ? ? ? ?(* make sure we got what we wanted *) diff --git a/m3-sys/m3front/src/misc/Scanner.m3 b/m3-sys/m3front/src/misc/Scanner.m3 index 7470374..e1dc024 100644 --- a/m3-sys/m3front/src/misc/Scanner.m3 +++ b/m3-sys/m3front/src/misc/Scanner.m3 @@ -228,13 +228,16 @@ PROCEDURE Here (VAR file: TEXT; ?VAR line: INTEGER) = ? ?BEGIN ? ? ?file := files [offset DIV MaxLines]; ? ? ?line := offset MOD MaxLines; + ? ?IF Target.ReduceTargetVariation THEN + ? ? ?file := Host.FileTail(file); + ? ?END; ? ?END Here; ? ?PROCEDURE LocalHere (VAR file: TEXT; ?VAR line: INTEGER) = ? ?VAR fnum := offset DIV MaxLines; ? ?BEGIN ? ? ?IF (local_files[fnum] = NIL) THEN - ? ? ?local_files[fnum] := files[fnum]; + ? ? ?local_files[fnum] := Host.FileTail(files[fnum]); ? ? ?END; ? ? ?file := local_files [fnum]; ? ? ?line := offset MOD MaxLines; diff --git a/m3-sys/m3front/src/values/Module.m3 b/m3-sys/m3front/src/values/Module.m3 index a085eab..576c857 100644 --- a/m3-sys/m3front/src/values/Module.m3 +++ b/m3-sys/m3front/src/values/Module.m3 @@ -421,7 +421,7 @@ PROCEDURE PushGeneric (t: T; ?VAR rd: File.T): M3ID.T = ? ? ?END; ? ? ? ?(* build a synthetic file name & start reading *) - ? ?filename := old_filename & " => " & filename; + ? ?filename := Host.FileTail(old_filename) & " => " & filename; ? ? ?Scanner.Push (filename, rd, is_main := Scanner.in_main); ? ? ?t.genericFile := filename; ? diff --git a/m3-sys/m3middle/src/Target.i3 b/m3-sys/m3middle/src/Target.i3 index fe198d2..cd7acee 100644 --- a/m3-sys/m3middle/src/Target.i3 +++ b/m3-sys/m3middle/src/Target.i3 @@ -529,4 +529,8 @@ VAR (*CONST*) ? ? ? test for nested procedures passed as parameters must be more ? ? ? elaborate (e.g. HPPA). *) ? + (* This removes some unnecessary target variation in the output, + ?* such as current working directory in debug output. *) + ReduceTargetVariation: BOOLEAN; + ?END Target. diff --git a/scripts/python/pylib.py b/scripts/python/pylib.py index 73e622f..487ad17 100755 --- a/scripts/python/pylib.py +++ b/scripts/python/pylib.py @@ -388,7 +388,7 @@ def _GetAllTargets(): ? ?_CBackend = "c" in sys.argv or "C" in sys.argv ?_BuildDirC = ["", "c"][_CBackend] -_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why"] +_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why", "reduce-target-variation", "reducetargetvariation"] ?_SkipGccFlags = ["nogcc", "skipgcc", "omitgcc"] ?_PossiblePylibFlags = ["noclean", "nocleangcc", "c", "C"] + _SkipGccFlags + _PossibleCm3Flags ?Thank you, ?- Jay From rodney_bates at lcwb.coop Mon Jul 4 16:55:38 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 04 Jul 2016 09:55:38 -0500 Subject: [M3devel] defaulting to parallel backend? In-Reply-To: References: Message-ID: <577A78EA.5000600@lcwb.coop> I prefer to use one fewer than the number of processors, so I can web browse, read email, review code, etc. while waiting for a compile. The compiler can saturate them all, making any other stuff very slow. On 07/04/2016 03:45 AM, Jay K wrote: > I suggest we make parallel backend the default. > > You can turn it off with -pb 1. > > Turn it "up" with -pb > > And the default is a thread per processor. > This will be kernel specific code, but we can get far with > > 1) Win32 GetSystemInfo, if applicable (e.g. C backend) > 2) sysconf(_SC_NPROCESSORS_ONLN) > > > and I'll research the various OSes (Darwin, Linux, *BSD, Solaris, and possibly AIX, HP-UX, Irix, Mingw, Cygwin). > ok? > > I could really use the speed boost and multi-processor machines are overwhelmingly common now (though easily avoided with VMs). > > > - Jay > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Mon Jul 4 16:58:32 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 04 Jul 2016 09:58:32 -0500 Subject: [M3devel] cm3 -reduce-target-variation / cm3cg -freduce-target-variation? In-Reply-To: References: Message-ID: <577A7998.1030907@lcwb.coop> On 07/04/2016 03:53 AM, Jay K wrote: > I guess I should learn to use branches and/or pull requests. > > > > This adds cm3 -reduce-target-variation and cm3cg -freduce-target-variation. > > > This meant for temporary development purposes. > No user would use them. > The point is to make e..g I386_FREEBSD and I386_NETBSD demonstrably identical, or at least closer to it. > > > Ok to commit? > If nothing happens unless these switches are explicitly supplied, it should have no effect on anybody else, so OK. > > Maybe name the two switches the same? Or something else? > > > Maybe be less aggressive? > i.e. even ../src/foo.m3 is getting changed to foo.m3. > The real target is ../I386_DARWIN/foo.m3 => foo.m3 > > > > diff --git a/m3-sys/cm3/src/Main.m3 b/m3-sys/cm3/src/Main.m3 > index 5d753e6..61b3f18 100644 > --- a/m3-sys/cm3/src/Main.m3 > +++ b/m3-sys/cm3/src/Main.m3 > @@ -5,7 +5,7 @@ MODULE Main; > > IMPORT M3Timers, Pathname, Process, Quake; > IMPORT RTCollector, RTParams, RTutils, Thread, Wr; > -IMPORT TextTextTbl; > +IMPORT TextTextTbl, Target; > > IMPORT Builder, Dirs, M3Build, M3Options, Makefile, Msg, Utils, WebFile; > IMPORT MxConfig(*, M3Config, CMKey, CMCurrent *); > @@ -18,6 +18,8 @@ VAR > build_dir : TEXT := NIL; > mach : Quake.Machine := NIL; > > +CONST BoolToText = ARRAY BOOLEAN OF TEXT{"FALSE", "TRUE"}; > + > PROCEDURE DefineIfNotDefined (qmachine: Quake.Machine; > symbol, value: TEXT) RAISES {Quake.Error} = > BEGIN > @@ -73,6 +75,7 @@ VAR defs: TextTextTbl.T; > (* DefineIfNotDefined (mach, "THREAD_LIBRARY", Version.ThreadLibrary); *) > (* DefineIfNotDefined (mach, "WINDOW_LIBRARY", Version.WindowLibrary); *) > DefineIfNotDefined (mach, "WORD_SIZE", MxConfig.HOST_WORD_SIZE); > + DefineIfNotDefined (mach, "REDUCE_TARGET_VARIATION", BoolToText[Target.ReduceTargetVariation]); > > (* Even if the config file overrides the defaults, such as to do > a cross build, the host characteristics are still available. *) > diff --git a/m3-sys/cm3/src/Makefile.m3 b/m3-sys/cm3/src/Makefile.m3 > index 64e169e..8b12320 100644 > --- a/m3-sys/cm3/src/Makefile.m3 > +++ b/m3-sys/cm3/src/Makefile.m3 > @@ -5,7 +5,7 @@ MODULE Makefile; > > IMPORT FS, M3File, M3Timers, OSError, Params, Process, Text, Thread, Wr; > IMPORT Arg, M3Build, M3Options, M3Path, Msg, Utils, TextSeq, TextTextTbl; > -IMPORT MxConfig, Dirs, Version; > +IMPORT MxConfig, Dirs, Version, Target; > > TYPE > NK = M3Path.Kind; > @@ -267,6 +267,9 @@ PROCEDURE ConvertOption (VAR s: State; arg: TEXT; arg_len: INTEGER) > | 'r' => IF Text.Equal(arg, "-realclean") THEN > ok := TRUE; (* mode set during the pre-scan *) > s.found_work := TRUE; > + ELSIF Text.Equal(arg, "-reduce-target-variation") THEN > + ok := TRUE; > + Target.ReduceTargetVariation := TRUE; > END; > > | 's' => IF Text.Equal (arg, "-silent") THEN > @@ -707,6 +710,9 @@ CONST > " -group-writable \"", > " -pb allow parallelism in running back-end (experimental)", > " -no-m3ship-resolution use quake variables in .M3SHIP (experimental)", > + " -reduce-target-variation omit target in some minor places such as", > + " current working directory in debug information", > + " for internal development purposes (showing target equivalence)", > "", > "environment variables:", > " M3CONFIG platform dependent configuration file to use (cm3.cfg)", > diff --git a/m3-sys/cminstall/src/config-no-install/cm3cfg.common b/m3-sys/cminstall/src/config-no-install/cm3cfg.common > index 7334252..9806c3c 100644 > --- a/m3-sys/cminstall/src/config-no-install/cm3cfg.common > +++ b/m3-sys/cminstall/src/config-no-install/cm3cfg.common > @@ -460,6 +460,12 @@ proc GetM3Back() is > end end > > m3back = "@" & m3back & "cm3cg " & GetM3BackFlags() > + > + if defined ("REDUCE_TARGET_VARIATION") > + if REDUCE_TARGET_VARIATION > + m3back = m3back & " -freduce-target-variation" > + end > + end > return m3back > end > > diff --git a/m3-sys/m3back/src/M3C.m3 b/m3-sys/m3back/src/M3C.m3 > index f6740c7..22c6ade 100644 > --- a/m3-sys/m3back/src/M3C.m3 > +++ b/m3-sys/m3back/src/M3C.m3 > @@ -29,7 +29,7 @@ VAR CaseDefaultAssertFalse := FALSE; > > (* Taken together, these help debugging, as you get more lines in the > C and the error messages reference C line numbers *) > - CONST output_line_directives = TRUE; > + VAR output_line_directives := TRUE; > CONST output_extra_newlines = FALSE; > CONST inline_extract = FALSE; > > @@ -2197,7 +2197,9 @@ BEGIN > END; > > IF (*self.suppress_line_directive < 1 AND*) text_last_char = '\n' THEN > + IF NOT Target.ReduceTargetVariation THEN > Wr.PutText(self.c, self.line_directive); > + END; > self.width := 0; > self.last_char_was_newline := TRUE; > RETURN; > @@ -2205,7 +2207,9 @@ BEGIN > > IF Text.FindChar(text, '\n') # -1 THEN > self.width := 0; (* roughly *) > + IF NOT Target.ReduceTargetVariation THEN > Wr.PutText(self.c, self.nl_line_directive); > + END; > self.last_char_was_newline := TRUE; > RETURN; > END; > @@ -2217,11 +2221,13 @@ BEGIN > END; > > self.width := 0; > + IF NOT Target.ReduceTargetVariation THEN > IF self.last_char_was_newline THEN > Wr.PutText(self.c, self.line_directive); > ELSE > Wr.PutText(self.c, self.nl_line_directive); > END; > + END; > self.last_char_was_newline := TRUE; > END print; > > @@ -2339,9 +2345,10 @@ END set_error_handler; > PROCEDURE Prefix_Print(self: T; multipass: Multipass_t) = > BEGIN > self.comment("begin unit"); > + output_line_directives := output_line_directives AND NOT Target.ReduceTargetVariation; > + IF NOT Target.ReduceTargetVariation THEN > self.comment("M3_TARGET = ", Target.System_name); > - (* This is an unnecessary target-specific output. *) > - (* self.comment("M3_TARGET = ", Target.System_name); *) > + END; > self.comment("M3_WORDSIZE = ", IntToDec(Target.Word.size)); > self.static_link_id := M3ID.Add("_static_link"); > self.alloca_id := M3ID.Add("alloca"); > diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c > index dc52576..33f8844 100644 > --- a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c > +++ b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c > @@ -1065,6 +1065,9 @@ dbxout_init (const char *input_file_name) > labels. */ > ASM_GENERATE_INTERNAL_LABEL (ltext_label_name, "Ltext", 0); > > + /* Limit paths in debug output, to limit target variation. */ > + if (!reduce_target_variation) > + { > /* Put the current working directory in an N_SO symbol. */ > if (use_gnu_debug_info_extensions && !NO_DBX_MAIN_SOURCE_DIRECTORY) > { > @@ -1087,6 +1090,7 @@ dbxout_init (const char *input_file_name) > used_ltext_label_name = true; > #endif /* no DBX_OUTPUT_MAIN_SOURCE_DIRECTORY */ > } > + } > > mapped_name = remap_debug_filename (input_file_name); > #ifdef DBX_OUTPUT_MAIN_SOURCE_FILENAME > diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c > index 0da1021..0a48a65 100644 > --- a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c > +++ b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c > @@ -15450,19 +15450,20 @@ add_gnat_descriptive_type_attribute (dw_die_ref die, tree type, > static void > add_comp_dir_attribute (dw_die_ref die) > { > + /* Limit paths in debug output, to limit target variation. */ > + if (reduce_target_variation) > + return; > + > const char *wd = get_src_pwd (); > - char *wd1; > > if (wd == NULL) > return; > > if (DWARF2_DIR_SHOULD_END_WITH_SEPARATOR) > { > - int wdlen; > - > - wdlen = strlen (wd); > - wd1 = (char *) ggc_alloc_atomic (wdlen + 2); > - strcpy (wd1, wd); > + int const wdlen = (int)strlen (wd); > + char * const wd1 = (char *) ggc_alloc_atomic (wdlen + 2); > + memcpy (wd1, wd, wdlen); > wd1 [wdlen] = DIR_SEPARATOR; > wd1 [wdlen + 1] = 0; > wd = wd1; > diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c > index 63e4b92..d468632 100644 > --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c > +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c > @@ -217,11 +217,8 @@ const char * > get_src_pwd (void) > { > if (! src_pwd) > - { > - src_pwd = getpwd (); > - if (!src_pwd) > + if (reduce_target_variation || !(src_pwd = getpwd ())) > src_pwd = "."; > - } > > return src_pwd; > } > diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h > index 588cfdb..f4f7cc7 100644 > --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h > +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h > @@ -80,4 +80,6 @@ extern bool set_src_pwd (const char *); > extern HOST_WIDE_INT get_random_seed (bool); > extern const char *set_random_seed (const char *); > > +extern bool reduce_target_variation; > + > #endif /* ! GCC_TOPLEV_H */ > diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt > index 3bd0469..edfca96 100644 > --- a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt > +++ b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt > @@ -28,9 +28,6 @@ m3cg > Language > M3CG > > -y > -m3cg M3CG > - > fopcodes-trace > m3cg M3CG > Trace opcodes > @@ -59,10 +56,17 @@ ftypes-trace > m3cg M3CG > Trace types > > +freduce-target-variation > +m3cg M3CG > +Reduce target variation somewhat, such as by omitting current working > +directory from debug info. Many necessary target variations remain. > + > v > m3cg M3CG > +print version > > y > m3cg M3CG > +Trace opcodes > > ; This comment is to ensure we retain the blank line above. > diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c > index 03417ed..0cfa46a 100644 > --- a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c > +++ b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c > @@ -246,6 +246,7 @@ build_case_label (tree low_value, tree high_value, tree label_decl) > /*======================================================= OPTION HANDLING ===*/ > > static int option_trace_all; > +bool reduce_target_variation; > > /*===========================================================================*/ > > @@ -6364,6 +6365,10 @@ m3_handle_option (size_t code, PCSTR /*arg*/, int /*value*/) > case OPT_ftypes_trace: > option_trace_all += 1; > break; > + > + case OPT_freduce_target_variation: > + reduce_target_variation = true; > + break; > } > > return 1; > diff --git a/m3-sys/m3front/src/misc/Coverage.m3 b/m3-sys/m3front/src/misc/Coverage.m3 > index c04c902..73fff21 100644 > --- a/m3-sys/m3front/src/misc/Coverage.m3 > +++ b/m3-sys/m3front/src/misc/Coverage.m3 > @@ -77,8 +77,9 @@ PROCEDURE NoteProcedure (v: Value.T) = > PROCEDURE GenerateTables () = > VAR > nLines := MAX (0, maxLine - minLine) + 1; > + fname := Host.FileTail (Host.filename); > l_header := TLen (Header); > - l_fname := TLen (Host.filename); > + l_fname := TLen (fname); > l_trailer := TLen (Trailer); > size : INTEGER; > p : ProcHead; > @@ -124,10 +125,10 @@ PROCEDURE GenerateTables () = > (* CG.Init_int (size, Target.Integer.size, TInt.Zero, FALSE); *) > INC (size, Target.Integer.size); (*timestamp*) > > - CG.Init_intt (size, Target.Integer.size, Text.Length (Host.filename), FALSE); > + CG.Init_intt (size, Target.Integer.size, Text.Length (fname), FALSE); > INC (size, Target.Integer.size); (*fileLen*) > > - CG.Init_chars (size, Host.filename, FALSE); > + CG.Init_chars (size, fname, FALSE); > INC (size, l_fname * Target.Char.size); (*file*) > > CG.Init_intt (size, Target.Integer.size, minLine, FALSE); > diff --git a/m3-sys/m3front/src/misc/Host.i3 b/m3-sys/m3front/src/misc/Host.i3 > index c71489f..af6a89a 100644 > --- a/m3-sys/m3front/src/misc/Host.i3 > +++ b/m3-sys/m3front/src/misc/Host.i3 > @@ -75,4 +75,7 @@ PROCEDURE OpenUnit (name: M3ID.T; interface, generic: BOOLEAN; > > PROCEDURE CloseFile (rd: File.T); > > +PROCEDURE FileTail (path: TEXT): TEXT; > + (* returns the 'tail' of 'path' -- after any slashes or even spaces *) > + > END Host. > diff --git a/m3-sys/m3front/src/misc/Host.m3 b/m3-sys/m3front/src/misc/Host.m3 > index 962d3c6..79042e3 100644 > --- a/m3-sys/m3front/src/misc/Host.m3 > +++ b/m3-sys/m3front/src/misc/Host.m3 > @@ -9,7 +9,7 @@ > > MODULE Host; > > -IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler; > +IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler, Target; > > PROCEDURE Initialize (READONLY options: ARRAY OF TEXT): BOOLEAN = > BEGIN > @@ -192,5 +192,24 @@ PROCEDURE CloseFile (rd: File.T) = > END; > END CloseFile; > > +PROCEDURE FileTail (path: TEXT): TEXT = > + VAR c: CHAR; > + BEGIN > + IF NOT Target.ReduceTargetVariation THEN RETURN path; END; > + > + IF (path = NIL) THEN RETURN NIL END; > + > + (* search for the last slash or blank in the string *) > + FOR x := Text.Length (path) - 1 TO 0 BY -1 DO > + c := Text.GetChar (path, x); > + IF (c = '/') OR (c = ' ') OR (c = '\\') THEN > + RETURN Text.Sub (path, x+1); > + END; > + END; > + > + (* no slashes *) > + RETURN path; > + END FileTail; > + > BEGIN > END Host. > diff --git a/m3-sys/m3front/src/misc/M3Header.m3 b/m3-sys/m3front/src/misc/M3Header.m3 > index 1e4decf..877d77c 100644 > --- a/m3-sys/m3front/src/misc/M3Header.m3 > +++ b/m3-sys/m3front/src/misc/M3Header.m3 > @@ -104,7 +104,7 @@ PROCEDURE PushGeneric (VAR s: State) = > IF (s.generic = NIL) THEN s.failed := TRUE; RETURN; END; > > (* build a synthetic file name & start reading *) > - filename := old_filename & " => " & filename; > + filename := Host.FileTail(old_filename) & " => " & filename; > Scanner.Push (filename, s.generic, is_main := Scanner.in_main); > > (* make sure we got what we wanted *) > diff --git a/m3-sys/m3front/src/misc/Scanner.m3 b/m3-sys/m3front/src/misc/Scanner.m3 > index 7470374..e1dc024 100644 > --- a/m3-sys/m3front/src/misc/Scanner.m3 > +++ b/m3-sys/m3front/src/misc/Scanner.m3 > @@ -228,13 +228,16 @@ PROCEDURE Here (VAR file: TEXT; VAR line: INTEGER) = > BEGIN > file := files [offset DIV MaxLines]; > line := offset MOD MaxLines; > + IF Target.ReduceTargetVariation THEN > + file := Host.FileTail(file); > + END; > END Here; > > PROCEDURE LocalHere (VAR file: TEXT; VAR line: INTEGER) = > VAR fnum := offset DIV MaxLines; > BEGIN > IF (local_files[fnum] = NIL) THEN > - local_files[fnum] := files[fnum]; > + local_files[fnum] := Host.FileTail(files[fnum]); > END; > file := local_files [fnum]; > line := offset MOD MaxLines; > diff --git a/m3-sys/m3front/src/values/Module.m3 b/m3-sys/m3front/src/values/Module.m3 > index a085eab..576c857 100644 > --- a/m3-sys/m3front/src/values/Module.m3 > +++ b/m3-sys/m3front/src/values/Module.m3 > @@ -421,7 +421,7 @@ PROCEDURE PushGeneric (t: T; VAR rd: File.T): M3ID.T = > END; > > (* build a synthetic file name & start reading *) > - filename := old_filename & " => " & filename; > + filename := Host.FileTail(old_filename) & " => " & filename; > Scanner.Push (filename, rd, is_main := Scanner.in_main); > t.genericFile := filename; > > diff --git a/m3-sys/m3middle/src/Target.i3 b/m3-sys/m3middle/src/Target.i3 > index fe198d2..cd7acee 100644 > --- a/m3-sys/m3middle/src/Target.i3 > +++ b/m3-sys/m3middle/src/Target.i3 > @@ -529,4 +529,8 @@ VAR (*CONST*) > test for nested procedures passed as parameters must be more > elaborate (e.g. HPPA). *) > > + (* This removes some unnecessary target variation in the output, > + * such as current working directory in debug output. *) > + ReduceTargetVariation: BOOLEAN; > + > END Target. > diff --git a/scripts/python/pylib.py b/scripts/python/pylib.py > index 73e622f..487ad17 100755 > --- a/scripts/python/pylib.py > +++ b/scripts/python/pylib.py > @@ -388,7 +388,7 @@ def _GetAllTargets(): > > _CBackend = "c" in sys.argv or "C" in sys.argv > _BuildDirC = ["", "c"][_CBackend] > -_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why"] > +_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why", "reduce-target-variation", "reducetargetvariation"] > _SkipGccFlags = ["nogcc", "skipgcc", "omitgcc"] > _PossiblePylibFlags = ["noclean", "nocleangcc", "c", "C"] + _SkipGccFlags + _PossibleCm3Flags > > > Thank you, > - Jay > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Mon Jul 4 17:13:57 2016 From: jay.krell at cornell.edu (Jay K) Date: Mon, 4 Jul 2016 15:13:57 +0000 Subject: [M3devel] cm3 -reduce-target-variation / cm3cg -freduce-target-variation? In-Reply-To: <577A7998.1030907@lcwb.coop> References: , <577A7998.1030907@lcwb.coop> Message-ID: That is the intent. Some of this is actually a restoration of historical behavior -- except now still under a swtich, and arguably the historical behavior wasn't a good default. I'll remove the "f" on the cm3cg switch, so cm3 and cm3cg take the same parameters, which I think is a better convention than "-ffoo because cm3cg tends to use that." There is also a slight optimization in there and removal of a seemingly redundant cm3cg -v option -- it is specified twice. I'll check some build outputs to see if it redundant. ?- Jay ---------------------------------------- > Date: Mon, 4 Jul 2016 09:58:32 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com > Subject: Re: [M3devel] cm3 -reduce-target-variation / cm3cg -freduce-target-variation? > > > > On 07/04/2016 03:53 AM, Jay K wrote: >> I guess I should learn to use branches and/or pull requests. >> >> >> >> This adds cm3 -reduce-target-variation and cm3cg -freduce-target-variation. >> >> >> This meant for temporary development purposes. >> No user would use them. >> The point is to make e..g I386_FREEBSD and I386_NETBSD demonstrably identical, or at least closer to it. >> >> >> Ok to commit? >> > > If nothing happens unless these switches are explicitly supplied, it should > have no effect on anybody else, so OK. > >> >> Maybe name the two switches the same? Or something else? >> >> >> Maybe be less aggressive? >> i.e. even ../src/foo.m3 is getting changed to foo.m3. >> The real target is ../I386_DARWIN/foo.m3 => foo.m3 >> >> >> >> diff --git a/m3-sys/cm3/src/Main.m3 b/m3-sys/cm3/src/Main.m3 >> index 5d753e6..61b3f18 100644 >> --- a/m3-sys/cm3/src/Main.m3 >> +++ b/m3-sys/cm3/src/Main.m3 >> @@ -5,7 +5,7 @@ MODULE Main; >> >> IMPORT M3Timers, Pathname, Process, Quake; >> IMPORT RTCollector, RTParams, RTutils, Thread, Wr; >> -IMPORT TextTextTbl; >> +IMPORT TextTextTbl, Target; >> >> IMPORT Builder, Dirs, M3Build, M3Options, Makefile, Msg, Utils, WebFile; >> IMPORT MxConfig(*, M3Config, CMKey, CMCurrent *); >> @@ -18,6 +18,8 @@ VAR >> build_dir : TEXT := NIL; >> mach : Quake.Machine := NIL; >> >> +CONST BoolToText = ARRAY BOOLEAN OF TEXT{"FALSE", "TRUE"}; >> + >> PROCEDURE DefineIfNotDefined (qmachine: Quake.Machine; >> symbol, value: TEXT) RAISES {Quake.Error} = >> BEGIN >> @@ -73,6 +75,7 @@ VAR defs: TextTextTbl.T; >> (* DefineIfNotDefined (mach, "THREAD_LIBRARY", Version.ThreadLibrary); *) >> (* DefineIfNotDefined (mach, "WINDOW_LIBRARY", Version.WindowLibrary); *) >> DefineIfNotDefined (mach, "WORD_SIZE", MxConfig.HOST_WORD_SIZE); >> + DefineIfNotDefined (mach, "REDUCE_TARGET_VARIATION", BoolToText[Target.ReduceTargetVariation]); >> >> (* Even if the config file overrides the defaults, such as to do >> a cross build, the host characteristics are still available. *) >> diff --git a/m3-sys/cm3/src/Makefile.m3 b/m3-sys/cm3/src/Makefile.m3 >> index 64e169e..8b12320 100644 >> --- a/m3-sys/cm3/src/Makefile.m3 >> +++ b/m3-sys/cm3/src/Makefile.m3 >> @@ -5,7 +5,7 @@ MODULE Makefile; >> >> IMPORT FS, M3File, M3Timers, OSError, Params, Process, Text, Thread, Wr; >> IMPORT Arg, M3Build, M3Options, M3Path, Msg, Utils, TextSeq, TextTextTbl; >> -IMPORT MxConfig, Dirs, Version; >> +IMPORT MxConfig, Dirs, Version, Target; >> >> TYPE >> NK = M3Path.Kind; >> @@ -267,6 +267,9 @@ PROCEDURE ConvertOption (VAR s: State; arg: TEXT; arg_len: INTEGER) >> | 'r' => IF Text.Equal(arg, "-realclean") THEN >> ok := TRUE; (* mode set during the pre-scan *) >> s.found_work := TRUE; >> + ELSIF Text.Equal(arg, "-reduce-target-variation") THEN >> + ok := TRUE; >> + Target.ReduceTargetVariation := TRUE; >> END; >> >> | 's' => IF Text.Equal (arg, "-silent") THEN >> @@ -707,6 +710,9 @@ CONST >> " -group-writable \"", >> " -pb allow parallelism in running back-end (experimental)", >> " -no-m3ship-resolution use quake variables in .M3SHIP (experimental)", >> + " -reduce-target-variation omit target in some minor places such as", >> + " current working directory in debug information", >> + " for internal development purposes (showing target equivalence)", >> "", >> "environment variables:", >> " M3CONFIG platform dependent configuration file to use (cm3.cfg)", >> diff --git a/m3-sys/cminstall/src/config-no-install/cm3cfg.common b/m3-sys/cminstall/src/config-no-install/cm3cfg.common >> index 7334252..9806c3c 100644 >> --- a/m3-sys/cminstall/src/config-no-install/cm3cfg.common >> +++ b/m3-sys/cminstall/src/config-no-install/cm3cfg.common >> @@ -460,6 +460,12 @@ proc GetM3Back() is >> end end >> >> m3back = "@" & m3back & "cm3cg " & GetM3BackFlags() >> + >> + if defined ("REDUCE_TARGET_VARIATION") >> + if REDUCE_TARGET_VARIATION >> + m3back = m3back & " -freduce-target-variation" >> + end >> + end >> return m3back >> end >> >> diff --git a/m3-sys/m3back/src/M3C.m3 b/m3-sys/m3back/src/M3C.m3 >> index f6740c7..22c6ade 100644 >> --- a/m3-sys/m3back/src/M3C.m3 >> +++ b/m3-sys/m3back/src/M3C.m3 >> @@ -29,7 +29,7 @@ VAR CaseDefaultAssertFalse := FALSE; >> >> (* Taken together, these help debugging, as you get more lines in the >> C and the error messages reference C line numbers *) >> - CONST output_line_directives = TRUE; >> + VAR output_line_directives := TRUE; >> CONST output_extra_newlines = FALSE; >> CONST inline_extract = FALSE; >> >> @@ -2197,7 +2197,9 @@ BEGIN >> END; >> >> IF (*self.suppress_line_directive < 1 AND*) text_last_char = '\n' THEN >> + IF NOT Target.ReduceTargetVariation THEN >> Wr.PutText(self.c, self.line_directive); >> + END; >> self.width := 0; >> self.last_char_was_newline := TRUE; >> RETURN; >> @@ -2205,7 +2207,9 @@ BEGIN >> >> IF Text.FindChar(text, '\n') # -1 THEN >> self.width := 0; (* roughly *) >> + IF NOT Target.ReduceTargetVariation THEN >> Wr.PutText(self.c, self.nl_line_directive); >> + END; >> self.last_char_was_newline := TRUE; >> RETURN; >> END; >> @@ -2217,11 +2221,13 @@ BEGIN >> END; >> >> self.width := 0; >> + IF NOT Target.ReduceTargetVariation THEN >> IF self.last_char_was_newline THEN >> Wr.PutText(self.c, self.line_directive); >> ELSE >> Wr.PutText(self.c, self.nl_line_directive); >> END; >> + END; >> self.last_char_was_newline := TRUE; >> END print; >> >> @@ -2339,9 +2345,10 @@ END set_error_handler; >> PROCEDURE Prefix_Print(self: T; multipass: Multipass_t) = >> BEGIN >> self.comment("begin unit"); >> + output_line_directives := output_line_directives AND NOT Target.ReduceTargetVariation; >> + IF NOT Target.ReduceTargetVariation THEN >> self.comment("M3_TARGET = ", Target.System_name); >> - (* This is an unnecessary target-specific output. *) >> - (* self.comment("M3_TARGET = ", Target.System_name); *) >> + END; >> self.comment("M3_WORDSIZE = ", IntToDec(Target.Word.size)); >> self.static_link_id := M3ID.Add("_static_link"); >> self.alloca_id := M3ID.Add("alloca"); >> diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c >> index dc52576..33f8844 100644 >> --- a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c >> +++ b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c >> @@ -1065,6 +1065,9 @@ dbxout_init (const char *input_file_name) >> labels. */ >> ASM_GENERATE_INTERNAL_LABEL (ltext_label_name, "Ltext", 0); >> >> + /* Limit paths in debug output, to limit target variation. */ >> + if (!reduce_target_variation) >> + { >> /* Put the current working directory in an N_SO symbol. */ >> if (use_gnu_debug_info_extensions && !NO_DBX_MAIN_SOURCE_DIRECTORY) >> { >> @@ -1087,6 +1090,7 @@ dbxout_init (const char *input_file_name) >> used_ltext_label_name = true; >> #endif /* no DBX_OUTPUT_MAIN_SOURCE_DIRECTORY */ >> } >> + } >> >> mapped_name = remap_debug_filename (input_file_name); >> #ifdef DBX_OUTPUT_MAIN_SOURCE_FILENAME >> diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c >> index 0da1021..0a48a65 100644 >> --- a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c >> +++ b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c >> @@ -15450,19 +15450,20 @@ add_gnat_descriptive_type_attribute (dw_die_ref die, tree type, >> static void >> add_comp_dir_attribute (dw_die_ref die) >> { >> + /* Limit paths in debug output, to limit target variation. */ >> + if (reduce_target_variation) >> + return; >> + >> const char *wd = get_src_pwd (); >> - char *wd1; >> >> if (wd == NULL) >> return; >> >> if (DWARF2_DIR_SHOULD_END_WITH_SEPARATOR) >> { >> - int wdlen; >> - >> - wdlen = strlen (wd); >> - wd1 = (char *) ggc_alloc_atomic (wdlen + 2); >> - strcpy (wd1, wd); >> + int const wdlen = (int)strlen (wd); >> + char * const wd1 = (char *) ggc_alloc_atomic (wdlen + 2); >> + memcpy (wd1, wd, wdlen); >> wd1 [wdlen] = DIR_SEPARATOR; >> wd1 [wdlen + 1] = 0; >> wd = wd1; >> diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c >> index 63e4b92..d468632 100644 >> --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c >> +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c >> @@ -217,11 +217,8 @@ const char * >> get_src_pwd (void) >> { >> if (! src_pwd) >> - { >> - src_pwd = getpwd (); >> - if (!src_pwd) >> + if (reduce_target_variation || !(src_pwd = getpwd ())) >> src_pwd = "."; >> - } >> >> return src_pwd; >> } >> diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h >> index 588cfdb..f4f7cc7 100644 >> --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h >> +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h >> @@ -80,4 +80,6 @@ extern bool set_src_pwd (const char *); >> extern HOST_WIDE_INT get_random_seed (bool); >> extern const char *set_random_seed (const char *); >> >> +extern bool reduce_target_variation; >> + >> #endif /* ! GCC_TOPLEV_H */ >> diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt >> index 3bd0469..edfca96 100644 >> --- a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt >> +++ b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt >> @@ -28,9 +28,6 @@ m3cg >> Language >> M3CG >> >> -y >> -m3cg M3CG >> - >> fopcodes-trace >> m3cg M3CG >> Trace opcodes >> @@ -59,10 +56,17 @@ ftypes-trace >> m3cg M3CG >> Trace types >> >> +freduce-target-variation >> +m3cg M3CG >> +Reduce target variation somewhat, such as by omitting current working >> +directory from debug info. Many necessary target variations remain. >> + >> v >> m3cg M3CG >> +print version >> >> y >> m3cg M3CG >> +Trace opcodes >> >> ; This comment is to ensure we retain the blank line above. >> diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c >> index 03417ed..0cfa46a 100644 >> --- a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c >> +++ b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c >> @@ -246,6 +246,7 @@ build_case_label (tree low_value, tree high_value, tree label_decl) >> /*======================================================= OPTION HANDLING ===*/ >> >> static int option_trace_all; >> +bool reduce_target_variation; >> >> /*===========================================================================*/ >> >> @@ -6364,6 +6365,10 @@ m3_handle_option (size_t code, PCSTR /*arg*/, int /*value*/) >> case OPT_ftypes_trace: >> option_trace_all += 1; >> break; >> + >> + case OPT_freduce_target_variation: >> + reduce_target_variation = true; >> + break; >> } >> >> return 1; >> diff --git a/m3-sys/m3front/src/misc/Coverage.m3 b/m3-sys/m3front/src/misc/Coverage.m3 >> index c04c902..73fff21 100644 >> --- a/m3-sys/m3front/src/misc/Coverage.m3 >> +++ b/m3-sys/m3front/src/misc/Coverage.m3 >> @@ -77,8 +77,9 @@ PROCEDURE NoteProcedure (v: Value.T) = >> PROCEDURE GenerateTables () = >> VAR >> nLines := MAX (0, maxLine - minLine) + 1; >> + fname := Host.FileTail (Host.filename); >> l_header := TLen (Header); >> - l_fname := TLen (Host.filename); >> + l_fname := TLen (fname); >> l_trailer := TLen (Trailer); >> size : INTEGER; >> p : ProcHead; >> @@ -124,10 +125,10 @@ PROCEDURE GenerateTables () = >> (* CG.Init_int (size, Target.Integer.size, TInt.Zero, FALSE); *) >> INC (size, Target.Integer.size); (*timestamp*) >> >> - CG.Init_intt (size, Target.Integer.size, Text.Length (Host.filename), FALSE); >> + CG.Init_intt (size, Target.Integer.size, Text.Length (fname), FALSE); >> INC (size, Target.Integer.size); (*fileLen*) >> >> - CG.Init_chars (size, Host.filename, FALSE); >> + CG.Init_chars (size, fname, FALSE); >> INC (size, l_fname * Target.Char.size); (*file*) >> >> CG.Init_intt (size, Target.Integer.size, minLine, FALSE); >> diff --git a/m3-sys/m3front/src/misc/Host.i3 b/m3-sys/m3front/src/misc/Host.i3 >> index c71489f..af6a89a 100644 >> --- a/m3-sys/m3front/src/misc/Host.i3 >> +++ b/m3-sys/m3front/src/misc/Host.i3 >> @@ -75,4 +75,7 @@ PROCEDURE OpenUnit (name: M3ID.T; interface, generic: BOOLEAN; >> >> PROCEDURE CloseFile (rd: File.T); >> >> +PROCEDURE FileTail (path: TEXT): TEXT; >> + (* returns the 'tail' of 'path' -- after any slashes or even spaces *) >> + >> END Host. >> diff --git a/m3-sys/m3front/src/misc/Host.m3 b/m3-sys/m3front/src/misc/Host.m3 >> index 962d3c6..79042e3 100644 >> --- a/m3-sys/m3front/src/misc/Host.m3 >> +++ b/m3-sys/m3front/src/misc/Host.m3 >> @@ -9,7 +9,7 @@ >> >> MODULE Host; >> >> -IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler; >> +IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler, Target; >> >> PROCEDURE Initialize (READONLY options: ARRAY OF TEXT): BOOLEAN = >> BEGIN >> @@ -192,5 +192,24 @@ PROCEDURE CloseFile (rd: File.T) = >> END; >> END CloseFile; >> >> +PROCEDURE FileTail (path: TEXT): TEXT = >> + VAR c: CHAR; >> + BEGIN >> + IF NOT Target.ReduceTargetVariation THEN RETURN path; END; >> + >> + IF (path = NIL) THEN RETURN NIL END; >> + >> + (* search for the last slash or blank in the string *) >> + FOR x := Text.Length (path) - 1 TO 0 BY -1 DO >> + c := Text.GetChar (path, x); >> + IF (c = '/') OR (c = ' ') OR (c = '\\') THEN >> + RETURN Text.Sub (path, x+1); >> + END; >> + END; >> + >> + (* no slashes *) >> + RETURN path; >> + END FileTail; >> + >> BEGIN >> END Host. >> diff --git a/m3-sys/m3front/src/misc/M3Header.m3 b/m3-sys/m3front/src/misc/M3Header.m3 >> index 1e4decf..877d77c 100644 >> --- a/m3-sys/m3front/src/misc/M3Header.m3 >> +++ b/m3-sys/m3front/src/misc/M3Header.m3 >> @@ -104,7 +104,7 @@ PROCEDURE PushGeneric (VAR s: State) = >> IF (s.generic = NIL) THEN s.failed := TRUE; RETURN; END; >> >> (* build a synthetic file name & start reading *) >> - filename := old_filename & " => " & filename; >> + filename := Host.FileTail(old_filename) & " => " & filename; >> Scanner.Push (filename, s.generic, is_main := Scanner.in_main); >> >> (* make sure we got what we wanted *) >> diff --git a/m3-sys/m3front/src/misc/Scanner.m3 b/m3-sys/m3front/src/misc/Scanner.m3 >> index 7470374..e1dc024 100644 >> --- a/m3-sys/m3front/src/misc/Scanner.m3 >> +++ b/m3-sys/m3front/src/misc/Scanner.m3 >> @@ -228,13 +228,16 @@ PROCEDURE Here (VAR file: TEXT; VAR line: INTEGER) = >> BEGIN >> file := files [offset DIV MaxLines]; >> line := offset MOD MaxLines; >> + IF Target.ReduceTargetVariation THEN >> + file := Host.FileTail(file); >> + END; >> END Here; >> >> PROCEDURE LocalHere (VAR file: TEXT; VAR line: INTEGER) = >> VAR fnum := offset DIV MaxLines; >> BEGIN >> IF (local_files[fnum] = NIL) THEN >> - local_files[fnum] := files[fnum]; >> + local_files[fnum] := Host.FileTail(files[fnum]); >> END; >> file := local_files [fnum]; >> line := offset MOD MaxLines; >> diff --git a/m3-sys/m3front/src/values/Module.m3 b/m3-sys/m3front/src/values/Module.m3 >> index a085eab..576c857 100644 >> --- a/m3-sys/m3front/src/values/Module.m3 >> +++ b/m3-sys/m3front/src/values/Module.m3 >> @@ -421,7 +421,7 @@ PROCEDURE PushGeneric (t: T; VAR rd: File.T): M3ID.T = >> END; >> >> (* build a synthetic file name & start reading *) >> - filename := old_filename & " => " & filename; >> + filename := Host.FileTail(old_filename) & " => " & filename; >> Scanner.Push (filename, rd, is_main := Scanner.in_main); >> t.genericFile := filename; >> >> diff --git a/m3-sys/m3middle/src/Target.i3 b/m3-sys/m3middle/src/Target.i3 >> index fe198d2..cd7acee 100644 >> --- a/m3-sys/m3middle/src/Target.i3 >> +++ b/m3-sys/m3middle/src/Target.i3 >> @@ -529,4 +529,8 @@ VAR (*CONST*) >> test for nested procedures passed as parameters must be more >> elaborate (e.g. HPPA). *) >> >> + (* This removes some unnecessary target variation in the output, >> + * such as current working directory in debug output. *) >> + ReduceTargetVariation: BOOLEAN; >> + >> END Target. >> diff --git a/scripts/python/pylib.py b/scripts/python/pylib.py >> index 73e622f..487ad17 100755 >> --- a/scripts/python/pylib.py >> +++ b/scripts/python/pylib.py >> @@ -388,7 +388,7 @@ def _GetAllTargets(): >> >> _CBackend = "c" in sys.argv or "C" in sys.argv >> _BuildDirC = ["", "c"][_CBackend] >> -_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why"] >> +_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why", "reduce-target-variation", "reducetargetvariation"] >> _SkipGccFlags = ["nogcc", "skipgcc", "omitgcc"] >> _PossiblePylibFlags = ["noclean", "nocleangcc", "c", "C"] + _SkipGccFlags + _PossibleCm3Flags >> >> >> Thank you, >> - Jay >> >> >> _______________________________________________ >> M3devel mailing list >> M3devel at elegosoft.com >> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> > > -- > Rodney Bates > rodney.m.bates at acm.org > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel From jay.krell at cornell.edu Mon Jul 4 23:17:23 2016 From: jay.krell at cornell.edu (Jay K) Date: Mon, 4 Jul 2016 21:17:23 +0000 Subject: [M3devel] defaulting to parallel backend? In-Reply-To: <577A78EA.5000600@lcwb.coop> References: , <577A78EA.5000600@lcwb.coop> Message-ID: ok. Presumably something like: name it to match C++11 They call it: unsigned std::thread::hardware_concurrency(void); We should call it: PROCEDURE Thread.HardwareConcurrency(): INTEGER; OR PROCEDURE Thread.HardwareConcurrency(): Word.T; One could specify the following result for "unknown": <0, 0, 1 1 is nice and safe, but lossy vs. actually knowing it is 1. One could add another function for the known-ness boolean: PROCEDURE Thread.HardwareConcurrencyValid(): BOOLEAN; or PROCEDURE Thread.HardwareConcurrencyKnown(): BOOLEAN; "Unknown" I don't believe would really stem from the underlying implementation, there is no error path in the underlying operating systems, but be for platforms we don't have a port for, so is less interesting and just returning a sub-par 1 isn't so bad. ?- Jay ---------------------------------------- > Date: Mon, 4 Jul 2016 09:55:38 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com > Subject: Re: [M3devel] defaulting to parallel backend? > > I prefer to use one fewer than the number of processors, so I can > web browse, read email, review code, etc. while waiting for a > compile. The compiler can saturate them all, making any other > stuff very slow. > > On 07/04/2016 03:45 AM, Jay K wrote: >> I suggest we make parallel backend the default. >> >> You can turn it off with -pb 1. >> >> Turn it "up" with -pb >> >> And the default is a thread per processor. >> This will be kernel specific code, but we can get far with >> >> 1) Win32 GetSystemInfo, if applicable (e.g. C backend) >> 2) sysconf(_SC_NPROCESSORS_ONLN) >> >> >> and I'll research the various OSes (Darwin, Linux, *BSD, Solaris, and possibly AIX, HP-UX, Irix, Mingw, Cygwin). >> ok? >> >> I could really use the speed boost and multi-processor machines are overwhelmingly common now (though easily avoided with VMs). >> >> >> - Jay >> >> >> _______________________________________________ >> M3devel mailing list >> M3devel at elegosoft.com >> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> > > -- > Rodney Bates > rodney.m.bates at acm.org > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel From jay.krell at cornell.edu Tue Jul 5 09:54:14 2016 From: jay.krell at cornell.edu (Jay K) Date: Tue, 5 Jul 2016 07:54:14 +0000 Subject: [M3devel] some code from boost? Message-ID: I'd like to bring in some code from boost. To m3core -- Thread.HardwareConcurrency, i.e. the number of processors It is licensed like so: Boost Software License - Version 1.0 - August 17th, 2003 Permission is hereby granted, free of charge, to any person or organization obtaining a copy of the software and accompanying documentation covered by this license (the "Software") to use, reproduce, display, distribute, execute, and transmit the Software, and to prepare derivative works of the Software, and to permit third-parties to whom the Software is furnished to do so, all subject to the following: The copyright notices in the Software and this entire statement, including the above license grant, this restriction and the following disclaimer, must be included in all copies of the Software, in whole or in part, and all derivative works of the Software, unless such copies or derivative works are solely in the form of machine-executable object code generated by a source language processor. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. http://www.boost.org/LICENSE_1_0.txt) I find the license a little long and would rather not have more, but it is a good body of work overall. I would put the code in circa m3core/thread/boost/win32, m3core/thread/boost/posix or maybe m3core/thread/Common/ThreadBoostWin32.c ThreadBoostPosix.c ok? esp. on the licensing? It is just a few lines of code. They call this thread::hardware_concurrency and it is the number of CPUs or cores (this number is actually ambiguous these days, with packages, cores, and hyper-threading). They have another number called physical_concurrency we might want to also provide. They have implementations for at least the following the systems: ?Win32 GetSystemInfo ?glibc?get_nprocs ?sysconf(_SC_NPROCESSORS_ONLN) ?Apple and FreeBSD?sysctlbyname("hw.ncpu") ?HP-UX?pthread_num_processors_np For licensing or at least credit purposs, it should be two files -- Win32 and Posix. The code is C++ but can be changed to C, and for should should be. I suppose, if the license is too restrictive, I can keep it out of m3core and put it only in cm3? ?- Jay From jay.krell at cornell.edu Tue Jul 5 10:19:23 2016 From: jay.krell at cornell.edu (Jay K) Date: Tue, 5 Jul 2016 08:19:23 +0000 Subject: [M3devel] purported condition variable problems on Win32? Message-ID: https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV vs. http://www.cs.wustl.edu/~schmidt/win32-cv-1.html vs. https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain I wrote this: PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; alertable: BOOLEAN) RAISES {Alerted} = (* LL = m on entry and exit, but not for the duration * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp * NOTE that they merge the user lock and the condition lock. * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html * "3.3. The Generation Count Solution" *) Do we have the problems described in README.CV? I haven't looked through the ACE code to see to what extent they resemble solution 3.3, or if they changed as a result of this discussion -- which I admit I don't understandand haven't read closely. Spurious wakeups are ok, though should be minimized. I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. Thank you, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Jul 5 10:26:36 2016 From: jay.krell at cornell.edu (Jay K) Date: Tue, 5 Jul 2016 08:26:36 +0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: Message-ID: Here is another implementation to consider: https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Tue, 5 Jul 2016 08:19:23 +0000 Subject: [M3devel] purported condition variable problems on Win32? https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV vs. http://www.cs.wustl.edu/~schmidt/win32-cv-1.html vs. https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain I wrote this: PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; alertable: BOOLEAN) RAISES {Alerted} = (* LL = m on entry and exit, but not for the duration * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp * NOTE that they merge the user lock and the condition lock. * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html * "3.3. The Generation Count Solution" *) Do we have the problems described in README.CV? I haven't looked through the ACE code to see to what extent they resemble solution 3.3, or if they changed as a result of this discussion -- which I admit I don't understandand haven't read closely. Spurious wakeups are ok, though should be minimized. I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. Thank you, - Jay _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From lemming at henning-thielemann.de Tue Jul 5 18:34:54 2016 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Tue, 5 Jul 2016 18:34:54 +0200 (CEST) Subject: [M3devel] defaulting to parallel backend? In-Reply-To: References: Message-ID: On Mon, 4 Jul 2016, Jay K wrote: > I suggest we make parallel backend the default. Generally I do not like if a program eats up all computing power without being told so. E.g. my machine has 8 hyperthreads but only 4 cores. Running on 8 processes hardly gives maximum efficiency, most oftenly speed is maximal at 3 processes. I also like the idea to use the remaining computing power for different tasks. Please just leave the choice of the number of processes to the user. From lemming at henning-thielemann.de Tue Jul 5 18:36:45 2016 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Tue, 5 Jul 2016 18:36:45 +0200 (CEST) Subject: [M3devel] defaulting to parallel backend? In-Reply-To: <577A78EA.5000600@lcwb.coop> References: <577A78EA.5000600@lcwb.coop> Message-ID: On Mon, 4 Jul 2016, Rodney M. Bates wrote: > I prefer to use one fewer than the number of processors, so I can > web browse, read email, review code, etc. while waiting for a > compile. The compiler can saturate them all, making any other > stuff very slow. Slowdown can happen even earlier if the compilation is bound by disk access time. From jay.krell at cornell.edu Wed Jul 6 07:44:45 2016 From: jay.krell at cornell.edu (Jay K) Date: Wed, 6 Jul 2016 05:44:45 +0000 Subject: [M3devel] defaulting to parallel backend? In-Reply-To: References: , <577A78EA.5000600@lcwb.coop>, Message-ID: There is no good easy answer here. You really want everything on the computer to use all of the available resources, to service whatever the human wants done as fast as possible, while remaining responsive e.g. to requests to cancel or requests to increase the work load. This is largely the job of the kernel, but it is an impossible job. To some extent we are obligated to over subscribe, in order to provide the kernel more information. But, again, it is an impossible job. Besides it probably being NP complete to complete the submitted work as soon as possible, we also don't know the user's priorities. I could propose "pb 1" to mean run single threaded, putting the burden on you to give. But that is a little rude. Note that historically you didn't have as much flexibility here -- single core systems. Anyway, how about the default be half? Or 1 if <= 3 cores available? I seem to have 4 cores, so this will help me. I understand this is still not great, e.g. if you have 10 high priority single processors jobs, and one low priority Modula-3 build, Modula-3 will by default use more than you wish. And yes, I understand disk are another resource. And network -- disk can be on network. You want to operate at the "edge of resource exhaustion" in general. - Jay > Date: Tue, 5 Jul 2016 18:36:45 +0200 > From: lemming at henning-thielemann.de > To: rodney.m.bates at acm.org > Subject: Re: [M3devel] defaulting to parallel backend? > CC: m3devel at elegosoft.com > > > On Mon, 4 Jul 2016, Rodney M. Bates wrote: > > > I prefer to use one fewer than the number of processors, so I can > > web browse, read email, review code, etc. while waiting for a > > compile. The compiler can saturate them all, making any other > > stuff very slow. > > Slowdown can happen even earlier if the compilation is bound by disk > access time. > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Jul 7 11:24:08 2016 From: jay.krell at cornell.edu (Jay K) Date: Thu, 7 Jul 2016 09:24:08 +0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: , Message-ID: So...I do NOT understand all of this. However, comparing the three implementationsand attempting to understand ours,I am struck by the following - Our broadcast seems ok, but - our signal seems to wake everyone, and then...they race a bit, one will decide it is last, and reset the event, but every prior waiter is still woken. Why not merge the following chunks: WHILE (NOT alerted) AND (NOT waitDone) DO... 1 EnterCriticalSection(conditionLock); waitDone := (c.tickets # 0 AND c.counter # count); LeaveCriticalSection(conditionLock); END; (* WHILE *) IF waitDone THEN alerted := FALSE; END; m.acquire(); 2 EnterCriticalSection(conditionLock); DEC(c.waiters); IF waitDone THEN esp. here. DEC(c.tickets); lastWaiter := (c.tickets = 0); END; LeaveCriticalSection(conditionLock); That is, if we decrement tickets earlier within the first critical section, while we will still wake everyone, only one will decide waitDone and the rest will keep looping. A downside of this, perhaps, is that waking all for Broadcast might be a little slower. but more so, in both the "ptw32" (pthreads for win32) and Boost threads implementations, they seem to deal with this differently than us, and they each do about the same thing -- they use a counted semaphore. In the boost case, it appears they duplicate the semaphore for every notification generation, which seems expensive. Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. Or they can do their own reference counting? jdk7 seems to looke like jdk6. The code is gone in jdk8 and I can't easily find the delete in history. The lock merging jdk does probably helps here too. In fact they merge #1 and #2 above as a result. But they still initially wake all threads for signal, not just broadcast. There is also the problem that all the event waits are followed by EnterCriticalSection (or jdk facsimile). - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: RE: [M3devel] purported condition variable problems on Win32? Date: Tue, 5 Jul 2016 08:26:36 +0000 Here is another implementation to consider: https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Tue, 5 Jul 2016 08:19:23 +0000 Subject: [M3devel] purported condition variable problems on Win32? https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV vs. http://www.cs.wustl.edu/~schmidt/win32-cv-1.html vs. https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain I wrote this: PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; alertable: BOOLEAN) RAISES {Alerted} = (* LL = m on entry and exit, but not for the duration * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp * NOTE that they merge the user lock and the condition lock. * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html * "3.3. The Generation Count Solution" *) Do we have the problems described in README.CV? I haven't looked through the ACE code to see to what extent they resemble solution 3.3, or if they changed as a result of this discussion -- which I admit I don't understandand haven't read closely. Spurious wakeups are ok, though should be minimized. I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. Thank you, - Jay _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Wed Jul 13 20:26:59 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Wed, 13 Jul 2016 13:26:59 -0500 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: Message-ID: <578687F3.1010108@lcwb.coop> I've begun looking at this. On 07/05/2016 03:19 AM, Jay K wrote: > https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > As for spurious wakeups, the M3 Thread.i3 documentation of Signal explicitly allows this: "One or more threads waiting on c become eligible to run", whereas the posix pthread_cond_signal, according to schmidt, "notifies one thread waiting on condition variable cv". As I recall, these (the M3 semantics) agree with the original Brinch-Hansen condition variables. So, for implementing posix CVs, this would be a true bug, but not so for M3 Thread CVs. > vs. > > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > vs. > https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > I wrote this: > PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > alertable: BOOLEAN) RAISES {Alerted} = > (* LL = m on entry and exit, but not for the duration > * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > * NOTE that they merge the user lock and the condition lock. > * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > * "3.3. The Generation Count Solution" > *) > > Do we have the problems described in README.CV? > > I haven't looked through the ACE code to see > to what extent they resemble solution 3.3, or if they > changed as a result of this discussion -- which I admit I don't understand > and haven't read closely. > > > Spurious wakeups are ok, though should be minimized. > > I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > Thank you, > - Jay > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Fri Jul 1 04:34:02 2016 From: jay.krell at cornell.edu (Jay K) Date: Fri, 1 Jul 2016 02:34:02 +0000 Subject: [M3devel] slight IR variation among targets? Message-ID: This is both a question and an explanation of hopefully an uncoming chnage, if I figure it out. Most platforms are almost the same. For example I386_FREEBSD and I386_NETBSD. You want their IR (and possibly code, but here just the IR) to generally be the same, along as long they aren't printing their actual target name. And they mostly are. Here is a different for example: jair:libm3 jay$ diff -du I386_NETBSD/NullRd.mc.txt I386_FREEBSD/NullRd.mc.txt? --- I386_NETBSD/NullRd.mc.txt 2016-06-30 19:13:11.000000000 -0700 +++ I386_FREEBSD/NullRd.mc.txt 2016-06-30 19:13:23.000000000 -0700 @@ -44,7 +44,6 @@ ? declare_procedure NullRd__Length 1 Int.32 0 0 F * p.7 ? declare_param rd 4 4 Addr 34129018 F F 50 v.10 ? declare_local _result 4 4 Int.32 425470580 F F 50 v.11 - reveal_opaque 34129018 -885236436 ? declare_opaque 483796623 -1651526519 ? declare_proctype -2116580915 2 -915982019 2 0 ? declare_formal n -1746782238 @@ -85,6 +84,7 @@ ? declare_field closed 416 8 509158269 ? declare_field seekable 424 8 509158269 ? declare_field intermittent 432 8 509158269 + reveal_opaque 34129018 -885236436 ? # Init ? -----LINE 22 ?----- ? begin_procedure p.5 One line of the IR is moved. The meaning is the same. What causes this? Do we output sometimes in hash order? And target affects that? Or when buffers fill up, and target length can affect that? I'd like to remove these differences. If it is a buffer length matter, I might just expand the buffers, as systems have far far more memory than when this was all written. If it is hash ordering matter -- one should avoid outputting data in hash order. ?- Jay From jay.krell at cornell.edu Fri Jul 1 08:48:57 2016 From: jay.krell at cornell.edu (Jay K) Date: Fri, 1 Jul 2016 06:48:57 +0000 Subject: [M3devel] lang.opt with -y twice? Message-ID: any idea why -y is here twice? Seems redundant. https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt?rev=1.2;content-type=text%2Fplain -y is shorthand for trace all, I know, Tony's advise years ago, very useful ?- Jay From rodney_bates at lcwb.coop Fri Jul 1 17:01:51 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Fri, 01 Jul 2016 10:01:51 -0500 Subject: [M3devel] source paths to generics? In-Reply-To: References: , , , , , , , <57746905.2080007@lcwb.coop> , <5775407E.6000401@lcwb.coop> Message-ID: <577685DF.2050404@lcwb.coop> On 06/30/2016 11:11 AM, Jay K wrote: > I put in a bunch of calls to RTIO.PutText("I'm here 1"); RTIO.Flush(), RTIO.PutText("I'm here 2"); RTIO.Flush(). > > When I saw those I knew I missed it somehow and looked again. > > I still don't see the strings in the output though. > > - Jay > > The missing call to M3Front.ParseImports is at cm3/src/Builder.m3:2007. It's in my previous grep output, plain as day. I must have just missed it in all the noise. But I'm still pretty sure the "=>" ("1=>"?) inserted by M3Header.Parse will never find its way into any output. > > ---------------------------------------- >> Date: Thu, 30 Jun 2016 10:53:34 -0500 >> From: rodney_bates at lcwb.coop >> To: m3devel at elegosoft.com >> Subject: Re: [M3devel] source paths to generics? >> >> >> >> On 06/29/2016 11:49 PM, Jay K wrote: >>> I guess neither of us looked outside of m3front. >>> The code runs. Not clear if the strings can get output. >>> Maybe from asserts or Compiler.ThisFile in a generic? >>> I"ll try some more. >>> >> >> I started looking in all of m3-sys, since the different compiler >> packages there are linked together into cm3. Later, I >> looked in the entire cm3 repo and didn't see anything. >> In any case, since it's part of the compiler, it really would be >> strange to call it from outside m3-sys. >> >> But if it is executing, we must have missed something. How did you >> find it executing? In a debugger? Can you do a backtrace? >> >>> jair:m3core jay$ grep -i parseimp ../../m3-sys/cm3/src/Builder.m3 >>> ids := M3Front.ParseImports (source, s.m3env); >>> >>> >>> - Jay >>> >>> >>> ---------------------------------------- >>>> Date: Wed, 29 Jun 2016 19:34:13 -0500 >>>> From: rodney_bates at lcwb.coop >>>> To: m3devel at elegosoft.com >>>> Subject: Re: [M3devel] source paths to generics? >>>> >>>> Hmm. I poked around a bit, and it looks like there is some cruft here, >>>> left over from something. >>>> >>>> The only call I can find on M3Header.Parse is M3Front.m3:37. This is >>>> in M3Front.ParseImports. I can find no calls on that at all. (There >>>> are other procedures by this name.) >>>> >>>> Also, it appears M3Header.ParseImports actually collects the exports, the imports, >>>> the generic actuals. >>>> >>>> As for the concocted (by M3Header) name of the form => , >>>> this looks to be used only to initialize the scanner's file number while scanning >>>> up through the formals of the generic, but that is not used. >>>> >>>> On 06/29/2016 11:34 AM, Jay K wrote: >>>>> There is similar code in Module.m3 and that is the code producing >>>>> the data "I don't like". >>>>> >>>>> M3Header isn't dead but I haven't seen it run yet. >>>>> >>>>> I changed them from "=>" to "1=>" and "2=>" and looked >>>>> where those occur in the output .s files under -keep. >>>>> >>>>> This is kinda something I wish were easier, like more strings >>>>> need to be longer for easier finding w/o ambiguity (the flip >>>>> side of my context arguments!) >>>>> >>>>> As things stand, I can't check that in. >>>>> I suppose maybe with a CG.comment but nevermind. >>>>> >>>>> - Jay >>>>> >>>>> >>>>> >>>>> ---------------------------------------- >>>>>> From: jay.krell at cornell.edu >>>>>> To: m3devel at elegosoft.com >>>>>> Subject: RE: [M3devel] source paths to generics? >>>>>> Date: Wed, 29 Jun 2016 15:22:14 +0000 >>>>>> >>>>>> mfront/src/misc/M3Header.m3 has this: >>>>>> >>>>>> >>>>>> (* build a synthetic file name & start reading *) >>>>>> filename := old_filename & " => " & filename; >>>>>> Scanner.Push (filename, s.generic, is_main := Scanner.in_main); >>>>>> >>>>>> >>>>>> and instantiated generics aren't what I thought. >>>>>> I thought the build system or compiler did a textual >>>>>> substitution of the generic parameters into the entire implementation, >>>>>> and saved that to the file system, >>>>>> and had the assert/debug paths point to it. >>>>>> >>>>>> So could step through what looks exactly like an .m3 file. >>>>>> >>>>>> But it doesn't. The instantiated file is a little stub, like: >>>>>> >>>>>> jair:libm3 jay$ more I386_DARWIN/TextAtomTbl.m3 >>>>>> (*generated by m3build*) >>>>>> >>>>>> MODULE TextAtomTbl = Table (Text, Atom) END TextAtomTbl. >>>>>> >>>>>> >>>>>> so I have one or two proposals. >>>>>> >>>>>> 1) old_filename above contains the target, it looks like: >>>>>> >>>>>> "../I386_DARWIN/TextAtomTbl.m3 => ../src/table/Table.mg" >>>>>> or >>>>>> "../I386_DARWIN/TextAtomTbl.m3 => ../src/table" >>>>>> >>>>>> These both occur. >>>>>> I'm not sure what the second is, seems like a mistake. >>>>>> >>>>>> I suggest the first string in both pairs be changed to be the leaf element, like: >>>>>> >>>>>> "TextAtomTbl.m3 => ../src/table/Table.mg" >>>>>> or >>>>>> "TextAtomTbl.m3 => ../src/table" >>>>>> >>>>>> and maybe the second string in both cases should be as it is in the second pair. >>>>>> >>>>>> 2) Possibly it should reall just be: >>>>>> ../src/table/Table.mg >>>>>> >>>>>> and developer can step through that, knowing that it is a somewhat abstracted >>>>>> form of what is running >>>>>> >>>>>> I'm willing to do this under like: >>>>>> >>>>>> CONST TemporaryForTargetConvergence = TRUE >>>>>> >>>>>> or >>>>>> VAR TemporaryForTargetConvergence: BOOLEAN; >>>>>> >>>>>> and a command line switch, but I suspect it isn't really useful to anyone asis, >>>>>> so might as well remove target unconditionally. >>>>>> >>>>>> >>>>>> ? >>>>>> >>>>>> - Jay >>>>>> >>>>>> >>>>>> >>>>>> ---------------------------------------- >>>>>>> From: jay.krell at cornell.edu >>>>>>> To: m3devel at elegosoft.com >>>>>>> Date: Wed, 29 Jun 2016 09:44:51 +0000 >>>>>>> Subject: [M3devel] source paths to generics? >>>>>>> >>>>>>> I haven't fully verified this, but it appears for example if I debug a generic, or fail an assert in a generic, >>>>>>> the source file I am told about is the instantiated i3/m3 file. >>>>>>> >>>>>>> This isn't particularly useful for the programmer or convenient for the compiler, right? >>>>>>> >>>>>>> The programmer would rather see the .ig/.mg files, and this is easy to provide in the compiler? >>>>>>> I guess it is slightly easier in the compiler, but easy to do better? >>>>>>> >>>>>>> I should look into make it so? >>>>>>> >>>>>>> My real agenda is I want to see: >>>>>>> ../src/foo.mg >>>>>>> >>>>>>> instead of >>>>>>> I386_DARWIN/foo.m3 >>>>>>> >>>>>>> I don't want the target variation, but other points seem true also, right? >>>>>>> >>>>>>> Right? The line numbers match between them, and the generic syntax is so close to "normal" that a programmer not used to it won't be confused, right? >>>>>>> >>>>>>> I'll try to poke around more. >>>>>>> >>>>>>> - Jay >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> M3devel mailing list >>>>>>> M3devel at elegosoft.com >>>>>>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>>>>> >>>>> >>>>> _______________________________________________ >>>>> M3devel mailing list >>>>> M3devel at elegosoft.com >>>>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>>>> >>>> >>>> -- >>>> Rodney Bates >>>> rodney.m.bates at acm.org >>>> _______________________________________________ >>>> M3devel mailing list >>>> M3devel at elegosoft.com >>>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>> >>> _______________________________________________ >>> M3devel mailing list >>> M3devel at elegosoft.com >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>> >> >> -- >> Rodney Bates >> rodney.m.bates at acm.org >> _______________________________________________ >> M3devel mailing list >> M3devel at elegosoft.com >> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > -- Rodney Bates rodney.m.bates at acm.org From wagner at elegosoft.com Fri Jul 1 17:17:10 2016 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 1 Jul 2016 17:17:10 +0200 Subject: [M3devel] source paths to generics? In-Reply-To: <577685DF.2050404@lcwb.coop> References: <57746905.2080007@lcwb.coop> <5775407E.6000401@lcwb.coop> <577685DF.2050404@lcwb.coop> Message-ID: <20160701171710.add85dcbf03e0e3f1f24c451@elegosoft.com> On Fri, 01 Jul 2016 10:01:51 -0500 "Rodney M. Bates" wrote: > On 06/30/2016 11:11 AM, Jay K wrote: > > I put in a bunch of calls to RTIO.PutText("I'm here 1"); RTIO.Flush(), RTIO.PutText("I'm here 2"); RTIO.Flush(). > > > > When I saw those I knew I missed it somehow and looked again. > > > > I still don't see the strings in the output though. > > > > - Jay > > > > > > The missing call to M3Front.ParseImports is at cm3/src/Builder.m3:2007. > It's in my previous grep output, plain as day. > I must have just missed it in all the noise. > > But I'm still pretty sure the "=>" ("1=>"?) inserted by M3Header.Parse will > never find its way into any output. I'm sure there's lots of old and obsolete code in the packages, as CM3 was built upon the DEC SRC compiler sources. The principal programmer of the compiler and Reactor is easily found via LinkedIn: https://www.linkedin.com/in/billkalsow I'm not sure if he is willing to answer questions about his work, but if you run into a real problem, it might be worthwhile to try it. Olaf -- Olaf Wagner -- elego Software Solutions GmbH -- http://www.elegosoft.com Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From rodney_bates at lcwb.coop Fri Jul 1 20:24:48 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Fri, 01 Jul 2016 13:24:48 -0500 Subject: [M3devel] source paths to generics? In-Reply-To: <20160701171710.add85dcbf03e0e3f1f24c451@elegosoft.com> References: <57746905.2080007@lcwb.coop> <5775407E.6000401@lcwb.coop> <577685DF.2050404@lcwb.coop> <20160701171710.add85dcbf03e0e3f1f24c451@elegosoft.com> Message-ID: <5776B570.1050504@lcwb.coop> On 07/01/2016 10:17 AM, Olaf Wagner wrote: > On Fri, 01 Jul 2016 10:01:51 -0500 > "Rodney M. Bates" wrote: > >> On 06/30/2016 11:11 AM, Jay K wrote: >>> I put in a bunch of calls to RTIO.PutText("I'm here 1"); RTIO.Flush(), RTIO.PutText("I'm here 2"); RTIO.Flush(). >>> >>> When I saw those I knew I missed it somehow and looked again. >>> >>> I still don't see the strings in the output though. >>> >>> - Jay >>> >>> >> >> The missing call to M3Front.ParseImports is at cm3/src/Builder.m3:2007. >> It's in my previous grep output, plain as day. >> I must have just missed it in all the noise. >> >> But I'm still pretty sure the "=>" ("1=>"?) inserted by M3Header.Parse will >> never find its way into any output. > > I'm sure there's lots of old and obsolete code in the packages, as CM3 > was built upon the DEC SRC compiler sources. > I didn't make this clear, but, with the newly discovered call, my disparaging remarks about M3Header.Parse, etc. are now refuted. > The principal programmer of the compiler and Reactor is easily found > via LinkedIn: https://www.linkedin.com/in/billkalsow > > I'm not sure if he is willing to answer questions about his work, but > if you run into a real problem, it might be worthwhile to try it. > > Olaf > -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Fri Jul 1 20:43:20 2016 From: jay.krell at cornell.edu (Jay K) Date: Fri, 1 Jul 2016 18:43:20 +0000 Subject: [M3devel] source paths to generics? In-Reply-To: <5776B570.1050504@lcwb.coop> References: <57746905.2080007@lcwb.coop> <5775407E.6000401@lcwb.coop> <577685DF.2050404@lcwb.coop>, <20160701171710.add85dcbf03e0e3f1f24c451@elegosoft.com>, <5776B570.1050504@lcwb.coop> Message-ID: >> But I'm still pretty sure the "=>" ("1=>"?) inserted by M3Header.Parse will>> never find its way into any output. agreed.Anyway, I'm almost done with my minor change here. I'm leaning toward: cm3 -trim-target-variation will set Target.TrimTargetVariation will also a quake variable CM3_TRIM_TARGET_VARIATION maybe quake function backend() will use quake variable to pass it on to cm3cg probably Unless maybe builder code knows it is calling cm3cg and will it in options there i.e. don't intend to push this through llvm backends at this time. Used by M3C to turn off line directives entirely (vs. just commenting them out as it is willing to do) Used by M3C to turn off a comment as what is TARGET cm3cg -ftrim-target-variation used by dbxout.c and dwarf2out.c to omit current working directory It is a funny name and a funny behavior and of limited use. The exercise is somewhat useful to see how to plumb options through. Much target variation occurs, just that slightly unnecessary variation will not, and then targets that are nearly the same, will be closer to the same. In particular, it likely that there are approximately one-two ABI per architecture on Posix hosts, therefore the Posix targets are all the same -- NetBSD, OpenBSD, Solaris, Darwin, Linux, FreeBSD. Darwin will tend to vary because it assumes newer processors so favors SSE over x87. But still for AMD64, probably the same. Windows tends to have different ABIs so the IR is still the same, but the assembly is not. - Jay > Date: Fri, 1 Jul 2016 13:24:48 -0500 > From: rodney_bates at lcwb.coop > To: wagner at elegosoft.com; jay.krell at cornell.edu > Subject: Re: [M3devel] source paths to generics? > CC: m3devel at elegosoft.com > > > > On 07/01/2016 10:17 AM, Olaf Wagner wrote: > > On Fri, 01 Jul 2016 10:01:51 -0500 > > "Rodney M. Bates" wrote: > > > >> On 06/30/2016 11:11 AM, Jay K wrote: > >>> I put in a bunch of calls to RTIO.PutText("I'm here 1"); RTIO.Flush(), RTIO.PutText("I'm here 2"); RTIO.Flush(). > >>> > >>> When I saw those I knew I missed it somehow and looked again. > >>> > >>> I still don't see the strings in the output though. > >>> > >>> - Jay > >>> > >>> > >> > >> The missing call to M3Front.ParseImports is at cm3/src/Builder.m3:2007. > >> It's in my previous grep output, plain as day. > >> I must have just missed it in all the noise. > >> > >> But I'm still pretty sure the "=>" ("1=>"?) inserted by M3Header.Parse will > >> never find its way into any output. > > > > I'm sure there's lots of old and obsolete code in the packages, as CM3 > > was built upon the DEC SRC compiler sources. > > > > I didn't make this clear, but, with the newly discovered call, my > disparaging remarks about M3Header.Parse, etc. are now refuted. > > > The principal programmer of the compiler and Reactor is easily found > > via LinkedIn: https://www.linkedin.com/in/billkalsow > > > > I'm not sure if he is willing to answer questions about his work, but > > if you run into a real problem, it might be worthwhile to try it. > > > > Olaf > > > > -- > Rodney Bates > rodney.m.bates at acm.org > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Jul 4 10:45:23 2016 From: jay.krell at cornell.edu (Jay K) Date: Mon, 4 Jul 2016 08:45:23 +0000 Subject: [M3devel] defaulting to parallel backend? Message-ID: I suggest we make parallel backend the default. You can turn it off with -pb 1. Turn it "up" with -pb And the default is a thread per processor. This will be kernel specific code, but we can get far with ?1) Win32 GetSystemInfo, if applicable (e.g. C backend) ?2)?sysconf(_SC_NPROCESSORS_ONLN) and I'll research the various OSes (Darwin, Linux, *BSD, Solaris, and possibly AIX, HP-UX, Irix, Mingw, Cygwin). ok? I could really use the speed boost and multi-processor machines are overwhelmingly common now (though easily avoided with VMs). ?- Jay From jay.krell at cornell.edu Mon Jul 4 10:53:44 2016 From: jay.krell at cornell.edu (Jay K) Date: Mon, 4 Jul 2016 08:53:44 +0000 Subject: [M3devel] cm3 -reduce-target-variation / cm3cg -freduce-target-variation? Message-ID: I guess I should learn to use branches and/or pull requests. This adds cm3 -reduce-target-variation and cm3cg -freduce-target-variation. This meant for temporary development purposes. No user would use them. The point is to make e..g I386_FREEBSD and I386_NETBSD demonstrably identical, or at least closer to it. Ok to commit? Maybe name the two switches the same? Or something else? Maybe be less aggressive? ?i.e. even ../src/foo.m3 is getting changed to foo.m3. ?The real target is ../I386_DARWIN/foo.m3 => foo.m3? ? ? diff --git a/m3-sys/cm3/src/Main.m3 b/m3-sys/cm3/src/Main.m3 index 5d753e6..61b3f18 100644 --- a/m3-sys/cm3/src/Main.m3 +++ b/m3-sys/cm3/src/Main.m3 @@ -5,7 +5,7 @@ MODULE Main; ? ?IMPORT M3Timers, Pathname, Process, Quake; ?IMPORT RTCollector, RTParams, RTutils, Thread, Wr; -IMPORT TextTextTbl; +IMPORT TextTextTbl, Target; ? ?IMPORT Builder, Dirs, M3Build, M3Options, Makefile, Msg, Utils, WebFile; ?IMPORT MxConfig(*, M3Config, CMKey, CMCurrent *); @@ -18,6 +18,8 @@ VAR ? ?build_dir : TEXT ? ? ? ? ?:= NIL; ? ?mach ? ? ?: Quake.Machine := NIL; ? +CONST BoolToText = ARRAY BOOLEAN OF TEXT{"FALSE", "TRUE"}; + ?PROCEDURE DefineIfNotDefined (qmachine: Quake.Machine; ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?symbol, value: TEXT) RAISES {Quake.Error} = ?BEGIN @@ -73,6 +75,7 @@ VAR defs: TextTextTbl.T; ? ? ? ? ?(* DefineIfNotDefined (mach, "THREAD_LIBRARY", Version.ThreadLibrary); *) ? ? ? ? ?(* DefineIfNotDefined (mach, "WINDOW_LIBRARY", Version.WindowLibrary); *) ? ? ? ? ?DefineIfNotDefined (mach, "WORD_SIZE", MxConfig.HOST_WORD_SIZE); + ? ? ? ?DefineIfNotDefined (mach, "REDUCE_TARGET_VARIATION", BoolToText[Target.ReduceTargetVariation]); ? ? ? ? ? ?(* Even if the config file overrides the defaults, such as to do ? ? ? ? ? ? a cross build, the host characteristics are still available. *) diff --git a/m3-sys/cm3/src/Makefile.m3 b/m3-sys/cm3/src/Makefile.m3 index 64e169e..8b12320 100644 --- a/m3-sys/cm3/src/Makefile.m3 +++ b/m3-sys/cm3/src/Makefile.m3 @@ -5,7 +5,7 @@ MODULE Makefile; ? ?IMPORT FS, M3File, M3Timers, OSError, Params, Process, Text, Thread, Wr; ?IMPORT Arg, M3Build, M3Options, M3Path, Msg, Utils, TextSeq, TextTextTbl; -IMPORT MxConfig, Dirs, Version; +IMPORT MxConfig, Dirs, Version, Target; ? ?TYPE ? ?NK = M3Path.Kind; @@ -267,6 +267,9 @@ PROCEDURE ConvertOption (VAR s: State; ?arg: TEXT; ?arg_len: INTEGER) ? ? ?| 'r' => IF Text.Equal(arg, "-realclean") THEN ? ? ? ? ? ? ? ? ok := TRUE; ?(* mode set during the pre-scan *) ? ? ? ? ? ? ? ? s.found_work := TRUE; + ? ? ? ? ? ? ELSIF Text.Equal(arg, "-reduce-target-variation") THEN + ? ? ? ? ? ? ? ok := TRUE; + ? ? ? ? ? ? ? Target.ReduceTargetVariation := TRUE; ? ? ? ? ? ? ? END; ? ? ? ? ? ? ?? ? ? ?| 's' => IF Text.Equal (arg, "-silent") THEN @@ -707,6 +710,9 @@ CONST ? ? ?" ?-group-writable \"", ? ? ?" ?-pb ? ? ? ?allow parallelism in running back-end (experimental)", ? ? ?" ?-no-m3ship-resolution use quake variables in .M3SHIP (experimental)", + ? ?" ?-reduce-target-variation omit target in some minor places such as", + ? ?" ? ? ? ? ? ? ? ? ?current working directory in debug information", + ? ?" ? ? ? ? ? ? ? ? ?for internal development purposes (showing target equivalence)", ? ? ?"", ? ? ?"environment variables:", ? ? ?" ?M3CONFIG ? ? ? platform dependent configuration file to use (cm3.cfg)", diff --git a/m3-sys/cminstall/src/config-no-install/cm3cfg.common b/m3-sys/cminstall/src/config-no-install/cm3cfg.common index 7334252..9806c3c 100644 --- a/m3-sys/cminstall/src/config-no-install/cm3cfg.common +++ b/m3-sys/cminstall/src/config-no-install/cm3cfg.common @@ -460,6 +460,12 @@ proc GetM3Back() is ? ?end end ? ? ?m3back = "@" & m3back & "cm3cg " & GetM3BackFlags() + ? + ?if defined ("REDUCE_TARGET_VARIATION") + ? ?if REDUCE_TARGET_VARIATION + ? ? ?m3back = m3back & " -freduce-target-variation" + ? ?end + ?end ? ?return m3back ?end ? diff --git a/m3-sys/m3back/src/M3C.m3 b/m3-sys/m3back/src/M3C.m3 index f6740c7..22c6ade 100644 --- a/m3-sys/m3back/src/M3C.m3 +++ b/m3-sys/m3back/src/M3C.m3 @@ -29,7 +29,7 @@ VAR CaseDefaultAssertFalse := FALSE; ? ?(* Taken together, these help debugging, as you get more lines in the ? ? C and the error messages reference C line numbers *) - ?CONST output_line_directives = TRUE; + ?VAR ?output_line_directives := TRUE; ? ?CONST output_extra_newlines = FALSE; ? ?CONST inline_extract = FALSE; ? @@ -2197,7 +2197,9 @@ BEGIN ? ? ?END; ? ? ? ?IF (*self.suppress_line_directive < 1 AND*) text_last_char = '\n' THEN + ? ? ? ?IF NOT Target.ReduceTargetVariation THEN ? ? ? ? ? ?Wr.PutText(self.c, self.line_directive); + ? ? ? ?END; ? ? ? ? ?self.width := 0; ? ? ? ? ?self.last_char_was_newline := TRUE; ? ? ? ? ?RETURN; @@ -2205,7 +2207,9 @@ BEGIN ? ? ? ?IF Text.FindChar(text, '\n') # -1 THEN ? ? ? ? ?self.width := 0; (* roughly *) + ? ? ? ?IF NOT Target.ReduceTargetVariation THEN ? ? ? ? ? ?Wr.PutText(self.c, self.nl_line_directive); + ? ? ? ?END; ? ? ? ? ?self.last_char_was_newline := TRUE; ? ? ? ? ?RETURN; ? ? ?END; @@ -2217,11 +2221,13 @@ BEGIN ? ? ?END; ? ? ? ?self.width := 0; + ? ?IF NOT Target.ReduceTargetVariation THEN ? ? ? ?IF self.last_char_was_newline THEN ? ? ? ? ?Wr.PutText(self.c, self.line_directive); ? ? ? ?ELSE ? ? ? ? ?Wr.PutText(self.c, self.nl_line_directive); ? ? ? ?END; + ? ?END; ? ? ?self.last_char_was_newline := TRUE; ?END print; ? @@ -2339,9 +2345,10 @@ END set_error_handler; ?PROCEDURE Prefix_Print(self: T; multipass: Multipass_t) = ?BEGIN ? ? ?self.comment("begin unit"); + ? ?output_line_directives := output_line_directives AND NOT Target.ReduceTargetVariation; + ? ?IF NOT Target.ReduceTargetVariation THEN ? ? ? ?self.comment("M3_TARGET = ", Target.System_name); - ? ?(* This is an unnecessary target-specific output. *) - ? ?(* self.comment("M3_TARGET = ", Target.System_name); *) + ? ?END; ? ? ?self.comment("M3_WORDSIZE = ", IntToDec(Target.Word.size)); ? ? ?self.static_link_id := M3ID.Add("_static_link"); ? ? ?self.alloca_id := M3ID.Add("alloca"); diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c index dc52576..33f8844 100644 --- a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c +++ b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c @@ -1065,6 +1065,9 @@ dbxout_init (const char *input_file_name) ? ? ? labels. ?*/ ? ?ASM_GENERATE_INTERNAL_LABEL (ltext_label_name, "Ltext", 0); ? + ?/* Limit paths in debug output, to limit target variation. */ + ?if (!reduce_target_variation) + ?{ ? ? ?/* Put the current working directory in an N_SO symbol. ?*/ ? ? ?if (use_gnu_debug_info_extensions && !NO_DBX_MAIN_SOURCE_DIRECTORY) ? ? ?{ @@ -1087,6 +1090,7 @@ dbxout_init (const char *input_file_name) ? ? ? ?used_ltext_label_name = true; ?#endif /* no DBX_OUTPUT_MAIN_SOURCE_DIRECTORY */ ? ? ?} + ?} ? ? ?mapped_name = remap_debug_filename (input_file_name); ?#ifdef DBX_OUTPUT_MAIN_SOURCE_FILENAME diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c index 0da1021..0a48a65 100644 --- a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c +++ b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c @@ -15450,19 +15450,20 @@ add_gnat_descriptive_type_attribute (dw_die_ref die, tree type, ?static void ?add_comp_dir_attribute (dw_die_ref die) ?{ + ?/* Limit paths in debug output, to limit target variation. */ + ?if (reduce_target_variation) + ? ?return; + ? ?const char *wd = get_src_pwd (); - ?char *wd1; ? ? ?if (wd == NULL) ? ? ?return; ? ? ?if (DWARF2_DIR_SHOULD_END_WITH_SEPARATOR) ? ? ?{ - ? ? ?int wdlen; - - ? ? ?wdlen = strlen (wd); - ? ? ?wd1 = (char *) ggc_alloc_atomic (wdlen + 2); - ? ? ?strcpy (wd1, wd); + ? ? ?int const wdlen = (int)strlen (wd); + ? ? ?char * const wd1 = (char *) ggc_alloc_atomic (wdlen + 2); + ? ? ?memcpy (wd1, wd, wdlen); ? ? ? ?wd1 [wdlen] = DIR_SEPARATOR; ? ? ? ?wd1 [wdlen + 1] = 0; ? ? ? ?wd = wd1; diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c index 63e4b92..d468632 100644 --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c @@ -217,11 +217,8 @@ const char * ?get_src_pwd (void) ?{ ? ?if (! src_pwd) - ? ?{ - ? ? ?src_pwd = getpwd (); - ? ? ?if (!src_pwd) + ? ?if (reduce_target_variation || !(src_pwd = getpwd ())) ? ? ? ?src_pwd = "."; - ? ?} ? ? ? return src_pwd; ?} diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h index 588cfdb..f4f7cc7 100644 --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h @@ -80,4 +80,6 @@ extern bool set_src_pwd ? ? ? (const char *); ?extern HOST_WIDE_INT get_random_seed (bool); ?extern const char *set_random_seed (const char *); ? +extern bool reduce_target_variation; + ?#endif /* ! GCC_TOPLEV_H */ diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt index 3bd0469..edfca96 100644 --- a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt +++ b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt @@ -28,9 +28,6 @@ m3cg ?Language ?M3CG ? -y -m3cg M3CG - ?fopcodes-trace ?m3cg M3CG ?Trace opcodes @@ -59,10 +56,17 @@ ftypes-trace ?m3cg M3CG ?Trace types ? +freduce-target-variation +m3cg M3CG +Reduce target variation somewhat, such as by omitting current working +directory from debug info. Many necessary target variations remain. + ?v ?m3cg M3CG +print version ? ?y ?m3cg M3CG +Trace opcodes ? ?; This comment is to ensure we retain the blank line above. diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c index 03417ed..0cfa46a 100644 --- a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c +++ b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c @@ -246,6 +246,7 @@ build_case_label (tree low_value, tree high_value, tree label_decl) ?/*======================================================= OPTION HANDLING ===*/ ? ?static int option_trace_all; +bool reduce_target_variation; ? ?/*===========================================================================*/ ? @@ -6364,6 +6365,10 @@ m3_handle_option (size_t code, PCSTR /*arg*/, int /*value*/) ? ? ?case OPT_ftypes_trace: ? ? ? ?option_trace_all += 1; ? ? ? ?break; + ? ? ? + ? ?case OPT_freduce_target_variation: + ? ? ?reduce_target_variation = true; + ? ? ?break; ? ? ?} ? ? ?return 1; diff --git a/m3-sys/m3front/src/misc/Coverage.m3 b/m3-sys/m3front/src/misc/Coverage.m3 index c04c902..73fff21 100644 --- a/m3-sys/m3front/src/misc/Coverage.m3 +++ b/m3-sys/m3front/src/misc/Coverage.m3 @@ -77,8 +77,9 @@ PROCEDURE NoteProcedure (v: Value.T) = ?PROCEDURE GenerateTables () = ? ?VAR ? ? ?nLines ? ?:= MAX (0, maxLine - minLine) + 1; + ? ?fname ? ? := Host.FileTail (Host.filename); ? ? ?l_header ?:= TLen (Header); - ? ?l_fname ? := TLen (Host.filename); + ? ?l_fname ? := TLen (fname); ? ? ?l_trailer := TLen (Trailer); ? ? ?size ? ?: INTEGER; ? ? ?p ? ? ? : ProcHead; @@ -124,10 +125,10 @@ PROCEDURE GenerateTables () = ? ? ?(* CG.Init_int (size, Target.Integer.size, TInt.Zero, FALSE); *) ? ? ?INC (size, Target.Integer.size); ? ? ? ? ? ? (*timestamp*) ? - ? ?CG.Init_intt (size, Target.Integer.size, Text.Length (Host.filename), FALSE); + ? ?CG.Init_intt (size, Target.Integer.size, Text.Length (fname), FALSE); ? ? ?INC (size, Target.Integer.size); ? ? ? ? ? ? (*fileLen*) ? - ? ?CG.Init_chars (size, Host.filename, FALSE); + ? ?CG.Init_chars (size, fname, FALSE); ? ? ?INC (size, l_fname * Target.Char.size); ? ? ?(*file*) ? ? ? ?CG.Init_intt (size, Target.Integer.size, minLine, FALSE); diff --git a/m3-sys/m3front/src/misc/Host.i3 b/m3-sys/m3front/src/misc/Host.i3 index c71489f..af6a89a 100644 --- a/m3-sys/m3front/src/misc/Host.i3 +++ b/m3-sys/m3front/src/misc/Host.i3 @@ -75,4 +75,7 @@ PROCEDURE OpenUnit (name: M3ID.T; interface, generic: BOOLEAN; ? ?PROCEDURE CloseFile (rd: File.T); ? +PROCEDURE FileTail (path: TEXT): TEXT; + (* returns the 'tail' of 'path' -- after any slashes or even spaces *) + ?END Host. diff --git a/m3-sys/m3front/src/misc/Host.m3 b/m3-sys/m3front/src/misc/Host.m3 index 962d3c6..79042e3 100644 --- a/m3-sys/m3front/src/misc/Host.m3 +++ b/m3-sys/m3front/src/misc/Host.m3 @@ -9,7 +9,7 @@ ? ?MODULE Host; ? -IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler; +IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler, Target; ? ?PROCEDURE Initialize (READONLY options: ARRAY OF TEXT): BOOLEAN = ? ?BEGIN @@ -192,5 +192,24 @@ PROCEDURE CloseFile (rd: File.T) = ? ? ?END; ? ?END CloseFile; ? +PROCEDURE FileTail (path: TEXT): TEXT = + ?VAR c: CHAR; + ?BEGIN + ? ?IF NOT Target.ReduceTargetVariation THEN RETURN path; END; + + ? ?IF (path = NIL) THEN RETURN NIL END; + + ? ?(* search for the last slash or blank in the string *) + ? ?FOR x := Text.Length (path) - 1 TO 0 BY -1 DO + ? ? ?c := Text.GetChar (path, x); + ? ? ?IF (c = '/') OR (c = ' ') OR (c = '\\') THEN + ? ? ? ?RETURN Text.Sub (path, x+1); + ? ? ?END; + ? ?END; + + ? ?(* no slashes *) + ? ?RETURN path; + ?END FileTail; + ?BEGIN ?END Host. diff --git a/m3-sys/m3front/src/misc/M3Header.m3 b/m3-sys/m3front/src/misc/M3Header.m3 index 1e4decf..877d77c 100644 --- a/m3-sys/m3front/src/misc/M3Header.m3 +++ b/m3-sys/m3front/src/misc/M3Header.m3 @@ -104,7 +104,7 @@ PROCEDURE PushGeneric (VAR s: State) = ? ? ?IF (s.generic = NIL) THEN s.failed := TRUE; ?RETURN; END; ? ? ? ?(* build a synthetic file name & start reading *) - ? ?filename := old_filename & " => " & filename; + ? ?filename := Host.FileTail(old_filename) & " => " & filename; ? ? ?Scanner.Push (filename, s.generic, is_main := Scanner.in_main); ? ? ? ?(* make sure we got what we wanted *) diff --git a/m3-sys/m3front/src/misc/Scanner.m3 b/m3-sys/m3front/src/misc/Scanner.m3 index 7470374..e1dc024 100644 --- a/m3-sys/m3front/src/misc/Scanner.m3 +++ b/m3-sys/m3front/src/misc/Scanner.m3 @@ -228,13 +228,16 @@ PROCEDURE Here (VAR file: TEXT; ?VAR line: INTEGER) = ? ?BEGIN ? ? ?file := files [offset DIV MaxLines]; ? ? ?line := offset MOD MaxLines; + ? ?IF Target.ReduceTargetVariation THEN + ? ? ?file := Host.FileTail(file); + ? ?END; ? ?END Here; ? ?PROCEDURE LocalHere (VAR file: TEXT; ?VAR line: INTEGER) = ? ?VAR fnum := offset DIV MaxLines; ? ?BEGIN ? ? ?IF (local_files[fnum] = NIL) THEN - ? ? ?local_files[fnum] := files[fnum]; + ? ? ?local_files[fnum] := Host.FileTail(files[fnum]); ? ? ?END; ? ? ?file := local_files [fnum]; ? ? ?line := offset MOD MaxLines; diff --git a/m3-sys/m3front/src/values/Module.m3 b/m3-sys/m3front/src/values/Module.m3 index a085eab..576c857 100644 --- a/m3-sys/m3front/src/values/Module.m3 +++ b/m3-sys/m3front/src/values/Module.m3 @@ -421,7 +421,7 @@ PROCEDURE PushGeneric (t: T; ?VAR rd: File.T): M3ID.T = ? ? ?END; ? ? ? ?(* build a synthetic file name & start reading *) - ? ?filename := old_filename & " => " & filename; + ? ?filename := Host.FileTail(old_filename) & " => " & filename; ? ? ?Scanner.Push (filename, rd, is_main := Scanner.in_main); ? ? ?t.genericFile := filename; ? diff --git a/m3-sys/m3middle/src/Target.i3 b/m3-sys/m3middle/src/Target.i3 index fe198d2..cd7acee 100644 --- a/m3-sys/m3middle/src/Target.i3 +++ b/m3-sys/m3middle/src/Target.i3 @@ -529,4 +529,8 @@ VAR (*CONST*) ? ? ? test for nested procedures passed as parameters must be more ? ? ? elaborate (e.g. HPPA). *) ? + (* This removes some unnecessary target variation in the output, + ?* such as current working directory in debug output. *) + ReduceTargetVariation: BOOLEAN; + ?END Target. diff --git a/scripts/python/pylib.py b/scripts/python/pylib.py index 73e622f..487ad17 100755 --- a/scripts/python/pylib.py +++ b/scripts/python/pylib.py @@ -388,7 +388,7 @@ def _GetAllTargets(): ? ?_CBackend = "c" in sys.argv or "C" in sys.argv ?_BuildDirC = ["", "c"][_CBackend] -_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why"] +_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why", "reduce-target-variation", "reducetargetvariation"] ?_SkipGccFlags = ["nogcc", "skipgcc", "omitgcc"] ?_PossiblePylibFlags = ["noclean", "nocleangcc", "c", "C"] + _SkipGccFlags + _PossibleCm3Flags ?Thank you, ?- Jay From rodney_bates at lcwb.coop Mon Jul 4 16:55:38 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 04 Jul 2016 09:55:38 -0500 Subject: [M3devel] defaulting to parallel backend? In-Reply-To: References: Message-ID: <577A78EA.5000600@lcwb.coop> I prefer to use one fewer than the number of processors, so I can web browse, read email, review code, etc. while waiting for a compile. The compiler can saturate them all, making any other stuff very slow. On 07/04/2016 03:45 AM, Jay K wrote: > I suggest we make parallel backend the default. > > You can turn it off with -pb 1. > > Turn it "up" with -pb > > And the default is a thread per processor. > This will be kernel specific code, but we can get far with > > 1) Win32 GetSystemInfo, if applicable (e.g. C backend) > 2) sysconf(_SC_NPROCESSORS_ONLN) > > > and I'll research the various OSes (Darwin, Linux, *BSD, Solaris, and possibly AIX, HP-UX, Irix, Mingw, Cygwin). > ok? > > I could really use the speed boost and multi-processor machines are overwhelmingly common now (though easily avoided with VMs). > > > - Jay > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Mon Jul 4 16:58:32 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 04 Jul 2016 09:58:32 -0500 Subject: [M3devel] cm3 -reduce-target-variation / cm3cg -freduce-target-variation? In-Reply-To: References: Message-ID: <577A7998.1030907@lcwb.coop> On 07/04/2016 03:53 AM, Jay K wrote: > I guess I should learn to use branches and/or pull requests. > > > > This adds cm3 -reduce-target-variation and cm3cg -freduce-target-variation. > > > This meant for temporary development purposes. > No user would use them. > The point is to make e..g I386_FREEBSD and I386_NETBSD demonstrably identical, or at least closer to it. > > > Ok to commit? > If nothing happens unless these switches are explicitly supplied, it should have no effect on anybody else, so OK. > > Maybe name the two switches the same? Or something else? > > > Maybe be less aggressive? > i.e. even ../src/foo.m3 is getting changed to foo.m3. > The real target is ../I386_DARWIN/foo.m3 => foo.m3 > > > > diff --git a/m3-sys/cm3/src/Main.m3 b/m3-sys/cm3/src/Main.m3 > index 5d753e6..61b3f18 100644 > --- a/m3-sys/cm3/src/Main.m3 > +++ b/m3-sys/cm3/src/Main.m3 > @@ -5,7 +5,7 @@ MODULE Main; > > IMPORT M3Timers, Pathname, Process, Quake; > IMPORT RTCollector, RTParams, RTutils, Thread, Wr; > -IMPORT TextTextTbl; > +IMPORT TextTextTbl, Target; > > IMPORT Builder, Dirs, M3Build, M3Options, Makefile, Msg, Utils, WebFile; > IMPORT MxConfig(*, M3Config, CMKey, CMCurrent *); > @@ -18,6 +18,8 @@ VAR > build_dir : TEXT := NIL; > mach : Quake.Machine := NIL; > > +CONST BoolToText = ARRAY BOOLEAN OF TEXT{"FALSE", "TRUE"}; > + > PROCEDURE DefineIfNotDefined (qmachine: Quake.Machine; > symbol, value: TEXT) RAISES {Quake.Error} = > BEGIN > @@ -73,6 +75,7 @@ VAR defs: TextTextTbl.T; > (* DefineIfNotDefined (mach, "THREAD_LIBRARY", Version.ThreadLibrary); *) > (* DefineIfNotDefined (mach, "WINDOW_LIBRARY", Version.WindowLibrary); *) > DefineIfNotDefined (mach, "WORD_SIZE", MxConfig.HOST_WORD_SIZE); > + DefineIfNotDefined (mach, "REDUCE_TARGET_VARIATION", BoolToText[Target.ReduceTargetVariation]); > > (* Even if the config file overrides the defaults, such as to do > a cross build, the host characteristics are still available. *) > diff --git a/m3-sys/cm3/src/Makefile.m3 b/m3-sys/cm3/src/Makefile.m3 > index 64e169e..8b12320 100644 > --- a/m3-sys/cm3/src/Makefile.m3 > +++ b/m3-sys/cm3/src/Makefile.m3 > @@ -5,7 +5,7 @@ MODULE Makefile; > > IMPORT FS, M3File, M3Timers, OSError, Params, Process, Text, Thread, Wr; > IMPORT Arg, M3Build, M3Options, M3Path, Msg, Utils, TextSeq, TextTextTbl; > -IMPORT MxConfig, Dirs, Version; > +IMPORT MxConfig, Dirs, Version, Target; > > TYPE > NK = M3Path.Kind; > @@ -267,6 +267,9 @@ PROCEDURE ConvertOption (VAR s: State; arg: TEXT; arg_len: INTEGER) > | 'r' => IF Text.Equal(arg, "-realclean") THEN > ok := TRUE; (* mode set during the pre-scan *) > s.found_work := TRUE; > + ELSIF Text.Equal(arg, "-reduce-target-variation") THEN > + ok := TRUE; > + Target.ReduceTargetVariation := TRUE; > END; > > | 's' => IF Text.Equal (arg, "-silent") THEN > @@ -707,6 +710,9 @@ CONST > " -group-writable \"", > " -pb allow parallelism in running back-end (experimental)", > " -no-m3ship-resolution use quake variables in .M3SHIP (experimental)", > + " -reduce-target-variation omit target in some minor places such as", > + " current working directory in debug information", > + " for internal development purposes (showing target equivalence)", > "", > "environment variables:", > " M3CONFIG platform dependent configuration file to use (cm3.cfg)", > diff --git a/m3-sys/cminstall/src/config-no-install/cm3cfg.common b/m3-sys/cminstall/src/config-no-install/cm3cfg.common > index 7334252..9806c3c 100644 > --- a/m3-sys/cminstall/src/config-no-install/cm3cfg.common > +++ b/m3-sys/cminstall/src/config-no-install/cm3cfg.common > @@ -460,6 +460,12 @@ proc GetM3Back() is > end end > > m3back = "@" & m3back & "cm3cg " & GetM3BackFlags() > + > + if defined ("REDUCE_TARGET_VARIATION") > + if REDUCE_TARGET_VARIATION > + m3back = m3back & " -freduce-target-variation" > + end > + end > return m3back > end > > diff --git a/m3-sys/m3back/src/M3C.m3 b/m3-sys/m3back/src/M3C.m3 > index f6740c7..22c6ade 100644 > --- a/m3-sys/m3back/src/M3C.m3 > +++ b/m3-sys/m3back/src/M3C.m3 > @@ -29,7 +29,7 @@ VAR CaseDefaultAssertFalse := FALSE; > > (* Taken together, these help debugging, as you get more lines in the > C and the error messages reference C line numbers *) > - CONST output_line_directives = TRUE; > + VAR output_line_directives := TRUE; > CONST output_extra_newlines = FALSE; > CONST inline_extract = FALSE; > > @@ -2197,7 +2197,9 @@ BEGIN > END; > > IF (*self.suppress_line_directive < 1 AND*) text_last_char = '\n' THEN > + IF NOT Target.ReduceTargetVariation THEN > Wr.PutText(self.c, self.line_directive); > + END; > self.width := 0; > self.last_char_was_newline := TRUE; > RETURN; > @@ -2205,7 +2207,9 @@ BEGIN > > IF Text.FindChar(text, '\n') # -1 THEN > self.width := 0; (* roughly *) > + IF NOT Target.ReduceTargetVariation THEN > Wr.PutText(self.c, self.nl_line_directive); > + END; > self.last_char_was_newline := TRUE; > RETURN; > END; > @@ -2217,11 +2221,13 @@ BEGIN > END; > > self.width := 0; > + IF NOT Target.ReduceTargetVariation THEN > IF self.last_char_was_newline THEN > Wr.PutText(self.c, self.line_directive); > ELSE > Wr.PutText(self.c, self.nl_line_directive); > END; > + END; > self.last_char_was_newline := TRUE; > END print; > > @@ -2339,9 +2345,10 @@ END set_error_handler; > PROCEDURE Prefix_Print(self: T; multipass: Multipass_t) = > BEGIN > self.comment("begin unit"); > + output_line_directives := output_line_directives AND NOT Target.ReduceTargetVariation; > + IF NOT Target.ReduceTargetVariation THEN > self.comment("M3_TARGET = ", Target.System_name); > - (* This is an unnecessary target-specific output. *) > - (* self.comment("M3_TARGET = ", Target.System_name); *) > + END; > self.comment("M3_WORDSIZE = ", IntToDec(Target.Word.size)); > self.static_link_id := M3ID.Add("_static_link"); > self.alloca_id := M3ID.Add("alloca"); > diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c > index dc52576..33f8844 100644 > --- a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c > +++ b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c > @@ -1065,6 +1065,9 @@ dbxout_init (const char *input_file_name) > labels. */ > ASM_GENERATE_INTERNAL_LABEL (ltext_label_name, "Ltext", 0); > > + /* Limit paths in debug output, to limit target variation. */ > + if (!reduce_target_variation) > + { > /* Put the current working directory in an N_SO symbol. */ > if (use_gnu_debug_info_extensions && !NO_DBX_MAIN_SOURCE_DIRECTORY) > { > @@ -1087,6 +1090,7 @@ dbxout_init (const char *input_file_name) > used_ltext_label_name = true; > #endif /* no DBX_OUTPUT_MAIN_SOURCE_DIRECTORY */ > } > + } > > mapped_name = remap_debug_filename (input_file_name); > #ifdef DBX_OUTPUT_MAIN_SOURCE_FILENAME > diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c > index 0da1021..0a48a65 100644 > --- a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c > +++ b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c > @@ -15450,19 +15450,20 @@ add_gnat_descriptive_type_attribute (dw_die_ref die, tree type, > static void > add_comp_dir_attribute (dw_die_ref die) > { > + /* Limit paths in debug output, to limit target variation. */ > + if (reduce_target_variation) > + return; > + > const char *wd = get_src_pwd (); > - char *wd1; > > if (wd == NULL) > return; > > if (DWARF2_DIR_SHOULD_END_WITH_SEPARATOR) > { > - int wdlen; > - > - wdlen = strlen (wd); > - wd1 = (char *) ggc_alloc_atomic (wdlen + 2); > - strcpy (wd1, wd); > + int const wdlen = (int)strlen (wd); > + char * const wd1 = (char *) ggc_alloc_atomic (wdlen + 2); > + memcpy (wd1, wd, wdlen); > wd1 [wdlen] = DIR_SEPARATOR; > wd1 [wdlen + 1] = 0; > wd = wd1; > diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c > index 63e4b92..d468632 100644 > --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c > +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c > @@ -217,11 +217,8 @@ const char * > get_src_pwd (void) > { > if (! src_pwd) > - { > - src_pwd = getpwd (); > - if (!src_pwd) > + if (reduce_target_variation || !(src_pwd = getpwd ())) > src_pwd = "."; > - } > > return src_pwd; > } > diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h > index 588cfdb..f4f7cc7 100644 > --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h > +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h > @@ -80,4 +80,6 @@ extern bool set_src_pwd (const char *); > extern HOST_WIDE_INT get_random_seed (bool); > extern const char *set_random_seed (const char *); > > +extern bool reduce_target_variation; > + > #endif /* ! GCC_TOPLEV_H */ > diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt > index 3bd0469..edfca96 100644 > --- a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt > +++ b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt > @@ -28,9 +28,6 @@ m3cg > Language > M3CG > > -y > -m3cg M3CG > - > fopcodes-trace > m3cg M3CG > Trace opcodes > @@ -59,10 +56,17 @@ ftypes-trace > m3cg M3CG > Trace types > > +freduce-target-variation > +m3cg M3CG > +Reduce target variation somewhat, such as by omitting current working > +directory from debug info. Many necessary target variations remain. > + > v > m3cg M3CG > +print version > > y > m3cg M3CG > +Trace opcodes > > ; This comment is to ensure we retain the blank line above. > diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c > index 03417ed..0cfa46a 100644 > --- a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c > +++ b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c > @@ -246,6 +246,7 @@ build_case_label (tree low_value, tree high_value, tree label_decl) > /*======================================================= OPTION HANDLING ===*/ > > static int option_trace_all; > +bool reduce_target_variation; > > /*===========================================================================*/ > > @@ -6364,6 +6365,10 @@ m3_handle_option (size_t code, PCSTR /*arg*/, int /*value*/) > case OPT_ftypes_trace: > option_trace_all += 1; > break; > + > + case OPT_freduce_target_variation: > + reduce_target_variation = true; > + break; > } > > return 1; > diff --git a/m3-sys/m3front/src/misc/Coverage.m3 b/m3-sys/m3front/src/misc/Coverage.m3 > index c04c902..73fff21 100644 > --- a/m3-sys/m3front/src/misc/Coverage.m3 > +++ b/m3-sys/m3front/src/misc/Coverage.m3 > @@ -77,8 +77,9 @@ PROCEDURE NoteProcedure (v: Value.T) = > PROCEDURE GenerateTables () = > VAR > nLines := MAX (0, maxLine - minLine) + 1; > + fname := Host.FileTail (Host.filename); > l_header := TLen (Header); > - l_fname := TLen (Host.filename); > + l_fname := TLen (fname); > l_trailer := TLen (Trailer); > size : INTEGER; > p : ProcHead; > @@ -124,10 +125,10 @@ PROCEDURE GenerateTables () = > (* CG.Init_int (size, Target.Integer.size, TInt.Zero, FALSE); *) > INC (size, Target.Integer.size); (*timestamp*) > > - CG.Init_intt (size, Target.Integer.size, Text.Length (Host.filename), FALSE); > + CG.Init_intt (size, Target.Integer.size, Text.Length (fname), FALSE); > INC (size, Target.Integer.size); (*fileLen*) > > - CG.Init_chars (size, Host.filename, FALSE); > + CG.Init_chars (size, fname, FALSE); > INC (size, l_fname * Target.Char.size); (*file*) > > CG.Init_intt (size, Target.Integer.size, minLine, FALSE); > diff --git a/m3-sys/m3front/src/misc/Host.i3 b/m3-sys/m3front/src/misc/Host.i3 > index c71489f..af6a89a 100644 > --- a/m3-sys/m3front/src/misc/Host.i3 > +++ b/m3-sys/m3front/src/misc/Host.i3 > @@ -75,4 +75,7 @@ PROCEDURE OpenUnit (name: M3ID.T; interface, generic: BOOLEAN; > > PROCEDURE CloseFile (rd: File.T); > > +PROCEDURE FileTail (path: TEXT): TEXT; > + (* returns the 'tail' of 'path' -- after any slashes or even spaces *) > + > END Host. > diff --git a/m3-sys/m3front/src/misc/Host.m3 b/m3-sys/m3front/src/misc/Host.m3 > index 962d3c6..79042e3 100644 > --- a/m3-sys/m3front/src/misc/Host.m3 > +++ b/m3-sys/m3front/src/misc/Host.m3 > @@ -9,7 +9,7 @@ > > MODULE Host; > > -IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler; > +IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler, Target; > > PROCEDURE Initialize (READONLY options: ARRAY OF TEXT): BOOLEAN = > BEGIN > @@ -192,5 +192,24 @@ PROCEDURE CloseFile (rd: File.T) = > END; > END CloseFile; > > +PROCEDURE FileTail (path: TEXT): TEXT = > + VAR c: CHAR; > + BEGIN > + IF NOT Target.ReduceTargetVariation THEN RETURN path; END; > + > + IF (path = NIL) THEN RETURN NIL END; > + > + (* search for the last slash or blank in the string *) > + FOR x := Text.Length (path) - 1 TO 0 BY -1 DO > + c := Text.GetChar (path, x); > + IF (c = '/') OR (c = ' ') OR (c = '\\') THEN > + RETURN Text.Sub (path, x+1); > + END; > + END; > + > + (* no slashes *) > + RETURN path; > + END FileTail; > + > BEGIN > END Host. > diff --git a/m3-sys/m3front/src/misc/M3Header.m3 b/m3-sys/m3front/src/misc/M3Header.m3 > index 1e4decf..877d77c 100644 > --- a/m3-sys/m3front/src/misc/M3Header.m3 > +++ b/m3-sys/m3front/src/misc/M3Header.m3 > @@ -104,7 +104,7 @@ PROCEDURE PushGeneric (VAR s: State) = > IF (s.generic = NIL) THEN s.failed := TRUE; RETURN; END; > > (* build a synthetic file name & start reading *) > - filename := old_filename & " => " & filename; > + filename := Host.FileTail(old_filename) & " => " & filename; > Scanner.Push (filename, s.generic, is_main := Scanner.in_main); > > (* make sure we got what we wanted *) > diff --git a/m3-sys/m3front/src/misc/Scanner.m3 b/m3-sys/m3front/src/misc/Scanner.m3 > index 7470374..e1dc024 100644 > --- a/m3-sys/m3front/src/misc/Scanner.m3 > +++ b/m3-sys/m3front/src/misc/Scanner.m3 > @@ -228,13 +228,16 @@ PROCEDURE Here (VAR file: TEXT; VAR line: INTEGER) = > BEGIN > file := files [offset DIV MaxLines]; > line := offset MOD MaxLines; > + IF Target.ReduceTargetVariation THEN > + file := Host.FileTail(file); > + END; > END Here; > > PROCEDURE LocalHere (VAR file: TEXT; VAR line: INTEGER) = > VAR fnum := offset DIV MaxLines; > BEGIN > IF (local_files[fnum] = NIL) THEN > - local_files[fnum] := files[fnum]; > + local_files[fnum] := Host.FileTail(files[fnum]); > END; > file := local_files [fnum]; > line := offset MOD MaxLines; > diff --git a/m3-sys/m3front/src/values/Module.m3 b/m3-sys/m3front/src/values/Module.m3 > index a085eab..576c857 100644 > --- a/m3-sys/m3front/src/values/Module.m3 > +++ b/m3-sys/m3front/src/values/Module.m3 > @@ -421,7 +421,7 @@ PROCEDURE PushGeneric (t: T; VAR rd: File.T): M3ID.T = > END; > > (* build a synthetic file name & start reading *) > - filename := old_filename & " => " & filename; > + filename := Host.FileTail(old_filename) & " => " & filename; > Scanner.Push (filename, rd, is_main := Scanner.in_main); > t.genericFile := filename; > > diff --git a/m3-sys/m3middle/src/Target.i3 b/m3-sys/m3middle/src/Target.i3 > index fe198d2..cd7acee 100644 > --- a/m3-sys/m3middle/src/Target.i3 > +++ b/m3-sys/m3middle/src/Target.i3 > @@ -529,4 +529,8 @@ VAR (*CONST*) > test for nested procedures passed as parameters must be more > elaborate (e.g. HPPA). *) > > + (* This removes some unnecessary target variation in the output, > + * such as current working directory in debug output. *) > + ReduceTargetVariation: BOOLEAN; > + > END Target. > diff --git a/scripts/python/pylib.py b/scripts/python/pylib.py > index 73e622f..487ad17 100755 > --- a/scripts/python/pylib.py > +++ b/scripts/python/pylib.py > @@ -388,7 +388,7 @@ def _GetAllTargets(): > > _CBackend = "c" in sys.argv or "C" in sys.argv > _BuildDirC = ["", "c"][_CBackend] > -_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why"] > +_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why", "reduce-target-variation", "reducetargetvariation"] > _SkipGccFlags = ["nogcc", "skipgcc", "omitgcc"] > _PossiblePylibFlags = ["noclean", "nocleangcc", "c", "C"] + _SkipGccFlags + _PossibleCm3Flags > > > Thank you, > - Jay > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Mon Jul 4 17:13:57 2016 From: jay.krell at cornell.edu (Jay K) Date: Mon, 4 Jul 2016 15:13:57 +0000 Subject: [M3devel] cm3 -reduce-target-variation / cm3cg -freduce-target-variation? In-Reply-To: <577A7998.1030907@lcwb.coop> References: , <577A7998.1030907@lcwb.coop> Message-ID: That is the intent. Some of this is actually a restoration of historical behavior -- except now still under a swtich, and arguably the historical behavior wasn't a good default. I'll remove the "f" on the cm3cg switch, so cm3 and cm3cg take the same parameters, which I think is a better convention than "-ffoo because cm3cg tends to use that." There is also a slight optimization in there and removal of a seemingly redundant cm3cg -v option -- it is specified twice. I'll check some build outputs to see if it redundant. ?- Jay ---------------------------------------- > Date: Mon, 4 Jul 2016 09:58:32 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com > Subject: Re: [M3devel] cm3 -reduce-target-variation / cm3cg -freduce-target-variation? > > > > On 07/04/2016 03:53 AM, Jay K wrote: >> I guess I should learn to use branches and/or pull requests. >> >> >> >> This adds cm3 -reduce-target-variation and cm3cg -freduce-target-variation. >> >> >> This meant for temporary development purposes. >> No user would use them. >> The point is to make e..g I386_FREEBSD and I386_NETBSD demonstrably identical, or at least closer to it. >> >> >> Ok to commit? >> > > If nothing happens unless these switches are explicitly supplied, it should > have no effect on anybody else, so OK. > >> >> Maybe name the two switches the same? Or something else? >> >> >> Maybe be less aggressive? >> i.e. even ../src/foo.m3 is getting changed to foo.m3. >> The real target is ../I386_DARWIN/foo.m3 => foo.m3 >> >> >> >> diff --git a/m3-sys/cm3/src/Main.m3 b/m3-sys/cm3/src/Main.m3 >> index 5d753e6..61b3f18 100644 >> --- a/m3-sys/cm3/src/Main.m3 >> +++ b/m3-sys/cm3/src/Main.m3 >> @@ -5,7 +5,7 @@ MODULE Main; >> >> IMPORT M3Timers, Pathname, Process, Quake; >> IMPORT RTCollector, RTParams, RTutils, Thread, Wr; >> -IMPORT TextTextTbl; >> +IMPORT TextTextTbl, Target; >> >> IMPORT Builder, Dirs, M3Build, M3Options, Makefile, Msg, Utils, WebFile; >> IMPORT MxConfig(*, M3Config, CMKey, CMCurrent *); >> @@ -18,6 +18,8 @@ VAR >> build_dir : TEXT := NIL; >> mach : Quake.Machine := NIL; >> >> +CONST BoolToText = ARRAY BOOLEAN OF TEXT{"FALSE", "TRUE"}; >> + >> PROCEDURE DefineIfNotDefined (qmachine: Quake.Machine; >> symbol, value: TEXT) RAISES {Quake.Error} = >> BEGIN >> @@ -73,6 +75,7 @@ VAR defs: TextTextTbl.T; >> (* DefineIfNotDefined (mach, "THREAD_LIBRARY", Version.ThreadLibrary); *) >> (* DefineIfNotDefined (mach, "WINDOW_LIBRARY", Version.WindowLibrary); *) >> DefineIfNotDefined (mach, "WORD_SIZE", MxConfig.HOST_WORD_SIZE); >> + DefineIfNotDefined (mach, "REDUCE_TARGET_VARIATION", BoolToText[Target.ReduceTargetVariation]); >> >> (* Even if the config file overrides the defaults, such as to do >> a cross build, the host characteristics are still available. *) >> diff --git a/m3-sys/cm3/src/Makefile.m3 b/m3-sys/cm3/src/Makefile.m3 >> index 64e169e..8b12320 100644 >> --- a/m3-sys/cm3/src/Makefile.m3 >> +++ b/m3-sys/cm3/src/Makefile.m3 >> @@ -5,7 +5,7 @@ MODULE Makefile; >> >> IMPORT FS, M3File, M3Timers, OSError, Params, Process, Text, Thread, Wr; >> IMPORT Arg, M3Build, M3Options, M3Path, Msg, Utils, TextSeq, TextTextTbl; >> -IMPORT MxConfig, Dirs, Version; >> +IMPORT MxConfig, Dirs, Version, Target; >> >> TYPE >> NK = M3Path.Kind; >> @@ -267,6 +267,9 @@ PROCEDURE ConvertOption (VAR s: State; arg: TEXT; arg_len: INTEGER) >> | 'r' => IF Text.Equal(arg, "-realclean") THEN >> ok := TRUE; (* mode set during the pre-scan *) >> s.found_work := TRUE; >> + ELSIF Text.Equal(arg, "-reduce-target-variation") THEN >> + ok := TRUE; >> + Target.ReduceTargetVariation := TRUE; >> END; >> >> | 's' => IF Text.Equal (arg, "-silent") THEN >> @@ -707,6 +710,9 @@ CONST >> " -group-writable \"", >> " -pb allow parallelism in running back-end (experimental)", >> " -no-m3ship-resolution use quake variables in .M3SHIP (experimental)", >> + " -reduce-target-variation omit target in some minor places such as", >> + " current working directory in debug information", >> + " for internal development purposes (showing target equivalence)", >> "", >> "environment variables:", >> " M3CONFIG platform dependent configuration file to use (cm3.cfg)", >> diff --git a/m3-sys/cminstall/src/config-no-install/cm3cfg.common b/m3-sys/cminstall/src/config-no-install/cm3cfg.common >> index 7334252..9806c3c 100644 >> --- a/m3-sys/cminstall/src/config-no-install/cm3cfg.common >> +++ b/m3-sys/cminstall/src/config-no-install/cm3cfg.common >> @@ -460,6 +460,12 @@ proc GetM3Back() is >> end end >> >> m3back = "@" & m3back & "cm3cg " & GetM3BackFlags() >> + >> + if defined ("REDUCE_TARGET_VARIATION") >> + if REDUCE_TARGET_VARIATION >> + m3back = m3back & " -freduce-target-variation" >> + end >> + end >> return m3back >> end >> >> diff --git a/m3-sys/m3back/src/M3C.m3 b/m3-sys/m3back/src/M3C.m3 >> index f6740c7..22c6ade 100644 >> --- a/m3-sys/m3back/src/M3C.m3 >> +++ b/m3-sys/m3back/src/M3C.m3 >> @@ -29,7 +29,7 @@ VAR CaseDefaultAssertFalse := FALSE; >> >> (* Taken together, these help debugging, as you get more lines in the >> C and the error messages reference C line numbers *) >> - CONST output_line_directives = TRUE; >> + VAR output_line_directives := TRUE; >> CONST output_extra_newlines = FALSE; >> CONST inline_extract = FALSE; >> >> @@ -2197,7 +2197,9 @@ BEGIN >> END; >> >> IF (*self.suppress_line_directive < 1 AND*) text_last_char = '\n' THEN >> + IF NOT Target.ReduceTargetVariation THEN >> Wr.PutText(self.c, self.line_directive); >> + END; >> self.width := 0; >> self.last_char_was_newline := TRUE; >> RETURN; >> @@ -2205,7 +2207,9 @@ BEGIN >> >> IF Text.FindChar(text, '\n') # -1 THEN >> self.width := 0; (* roughly *) >> + IF NOT Target.ReduceTargetVariation THEN >> Wr.PutText(self.c, self.nl_line_directive); >> + END; >> self.last_char_was_newline := TRUE; >> RETURN; >> END; >> @@ -2217,11 +2221,13 @@ BEGIN >> END; >> >> self.width := 0; >> + IF NOT Target.ReduceTargetVariation THEN >> IF self.last_char_was_newline THEN >> Wr.PutText(self.c, self.line_directive); >> ELSE >> Wr.PutText(self.c, self.nl_line_directive); >> END; >> + END; >> self.last_char_was_newline := TRUE; >> END print; >> >> @@ -2339,9 +2345,10 @@ END set_error_handler; >> PROCEDURE Prefix_Print(self: T; multipass: Multipass_t) = >> BEGIN >> self.comment("begin unit"); >> + output_line_directives := output_line_directives AND NOT Target.ReduceTargetVariation; >> + IF NOT Target.ReduceTargetVariation THEN >> self.comment("M3_TARGET = ", Target.System_name); >> - (* This is an unnecessary target-specific output. *) >> - (* self.comment("M3_TARGET = ", Target.System_name); *) >> + END; >> self.comment("M3_WORDSIZE = ", IntToDec(Target.Word.size)); >> self.static_link_id := M3ID.Add("_static_link"); >> self.alloca_id := M3ID.Add("alloca"); >> diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c >> index dc52576..33f8844 100644 >> --- a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c >> +++ b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c >> @@ -1065,6 +1065,9 @@ dbxout_init (const char *input_file_name) >> labels. */ >> ASM_GENERATE_INTERNAL_LABEL (ltext_label_name, "Ltext", 0); >> >> + /* Limit paths in debug output, to limit target variation. */ >> + if (!reduce_target_variation) >> + { >> /* Put the current working directory in an N_SO symbol. */ >> if (use_gnu_debug_info_extensions && !NO_DBX_MAIN_SOURCE_DIRECTORY) >> { >> @@ -1087,6 +1090,7 @@ dbxout_init (const char *input_file_name) >> used_ltext_label_name = true; >> #endif /* no DBX_OUTPUT_MAIN_SOURCE_DIRECTORY */ >> } >> + } >> >> mapped_name = remap_debug_filename (input_file_name); >> #ifdef DBX_OUTPUT_MAIN_SOURCE_FILENAME >> diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c >> index 0da1021..0a48a65 100644 >> --- a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c >> +++ b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c >> @@ -15450,19 +15450,20 @@ add_gnat_descriptive_type_attribute (dw_die_ref die, tree type, >> static void >> add_comp_dir_attribute (dw_die_ref die) >> { >> + /* Limit paths in debug output, to limit target variation. */ >> + if (reduce_target_variation) >> + return; >> + >> const char *wd = get_src_pwd (); >> - char *wd1; >> >> if (wd == NULL) >> return; >> >> if (DWARF2_DIR_SHOULD_END_WITH_SEPARATOR) >> { >> - int wdlen; >> - >> - wdlen = strlen (wd); >> - wd1 = (char *) ggc_alloc_atomic (wdlen + 2); >> - strcpy (wd1, wd); >> + int const wdlen = (int)strlen (wd); >> + char * const wd1 = (char *) ggc_alloc_atomic (wdlen + 2); >> + memcpy (wd1, wd, wdlen); >> wd1 [wdlen] = DIR_SEPARATOR; >> wd1 [wdlen + 1] = 0; >> wd = wd1; >> diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c >> index 63e4b92..d468632 100644 >> --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c >> +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c >> @@ -217,11 +217,8 @@ const char * >> get_src_pwd (void) >> { >> if (! src_pwd) >> - { >> - src_pwd = getpwd (); >> - if (!src_pwd) >> + if (reduce_target_variation || !(src_pwd = getpwd ())) >> src_pwd = "."; >> - } >> >> return src_pwd; >> } >> diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h >> index 588cfdb..f4f7cc7 100644 >> --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h >> +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h >> @@ -80,4 +80,6 @@ extern bool set_src_pwd (const char *); >> extern HOST_WIDE_INT get_random_seed (bool); >> extern const char *set_random_seed (const char *); >> >> +extern bool reduce_target_variation; >> + >> #endif /* ! GCC_TOPLEV_H */ >> diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt >> index 3bd0469..edfca96 100644 >> --- a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt >> +++ b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt >> @@ -28,9 +28,6 @@ m3cg >> Language >> M3CG >> >> -y >> -m3cg M3CG >> - >> fopcodes-trace >> m3cg M3CG >> Trace opcodes >> @@ -59,10 +56,17 @@ ftypes-trace >> m3cg M3CG >> Trace types >> >> +freduce-target-variation >> +m3cg M3CG >> +Reduce target variation somewhat, such as by omitting current working >> +directory from debug info. Many necessary target variations remain. >> + >> v >> m3cg M3CG >> +print version >> >> y >> m3cg M3CG >> +Trace opcodes >> >> ; This comment is to ensure we retain the blank line above. >> diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c >> index 03417ed..0cfa46a 100644 >> --- a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c >> +++ b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c >> @@ -246,6 +246,7 @@ build_case_label (tree low_value, tree high_value, tree label_decl) >> /*======================================================= OPTION HANDLING ===*/ >> >> static int option_trace_all; >> +bool reduce_target_variation; >> >> /*===========================================================================*/ >> >> @@ -6364,6 +6365,10 @@ m3_handle_option (size_t code, PCSTR /*arg*/, int /*value*/) >> case OPT_ftypes_trace: >> option_trace_all += 1; >> break; >> + >> + case OPT_freduce_target_variation: >> + reduce_target_variation = true; >> + break; >> } >> >> return 1; >> diff --git a/m3-sys/m3front/src/misc/Coverage.m3 b/m3-sys/m3front/src/misc/Coverage.m3 >> index c04c902..73fff21 100644 >> --- a/m3-sys/m3front/src/misc/Coverage.m3 >> +++ b/m3-sys/m3front/src/misc/Coverage.m3 >> @@ -77,8 +77,9 @@ PROCEDURE NoteProcedure (v: Value.T) = >> PROCEDURE GenerateTables () = >> VAR >> nLines := MAX (0, maxLine - minLine) + 1; >> + fname := Host.FileTail (Host.filename); >> l_header := TLen (Header); >> - l_fname := TLen (Host.filename); >> + l_fname := TLen (fname); >> l_trailer := TLen (Trailer); >> size : INTEGER; >> p : ProcHead; >> @@ -124,10 +125,10 @@ PROCEDURE GenerateTables () = >> (* CG.Init_int (size, Target.Integer.size, TInt.Zero, FALSE); *) >> INC (size, Target.Integer.size); (*timestamp*) >> >> - CG.Init_intt (size, Target.Integer.size, Text.Length (Host.filename), FALSE); >> + CG.Init_intt (size, Target.Integer.size, Text.Length (fname), FALSE); >> INC (size, Target.Integer.size); (*fileLen*) >> >> - CG.Init_chars (size, Host.filename, FALSE); >> + CG.Init_chars (size, fname, FALSE); >> INC (size, l_fname * Target.Char.size); (*file*) >> >> CG.Init_intt (size, Target.Integer.size, minLine, FALSE); >> diff --git a/m3-sys/m3front/src/misc/Host.i3 b/m3-sys/m3front/src/misc/Host.i3 >> index c71489f..af6a89a 100644 >> --- a/m3-sys/m3front/src/misc/Host.i3 >> +++ b/m3-sys/m3front/src/misc/Host.i3 >> @@ -75,4 +75,7 @@ PROCEDURE OpenUnit (name: M3ID.T; interface, generic: BOOLEAN; >> >> PROCEDURE CloseFile (rd: File.T); >> >> +PROCEDURE FileTail (path: TEXT): TEXT; >> + (* returns the 'tail' of 'path' -- after any slashes or even spaces *) >> + >> END Host. >> diff --git a/m3-sys/m3front/src/misc/Host.m3 b/m3-sys/m3front/src/misc/Host.m3 >> index 962d3c6..79042e3 100644 >> --- a/m3-sys/m3front/src/misc/Host.m3 >> +++ b/m3-sys/m3front/src/misc/Host.m3 >> @@ -9,7 +9,7 @@ >> >> MODULE Host; >> >> -IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler; >> +IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler, Target; >> >> PROCEDURE Initialize (READONLY options: ARRAY OF TEXT): BOOLEAN = >> BEGIN >> @@ -192,5 +192,24 @@ PROCEDURE CloseFile (rd: File.T) = >> END; >> END CloseFile; >> >> +PROCEDURE FileTail (path: TEXT): TEXT = >> + VAR c: CHAR; >> + BEGIN >> + IF NOT Target.ReduceTargetVariation THEN RETURN path; END; >> + >> + IF (path = NIL) THEN RETURN NIL END; >> + >> + (* search for the last slash or blank in the string *) >> + FOR x := Text.Length (path) - 1 TO 0 BY -1 DO >> + c := Text.GetChar (path, x); >> + IF (c = '/') OR (c = ' ') OR (c = '\\') THEN >> + RETURN Text.Sub (path, x+1); >> + END; >> + END; >> + >> + (* no slashes *) >> + RETURN path; >> + END FileTail; >> + >> BEGIN >> END Host. >> diff --git a/m3-sys/m3front/src/misc/M3Header.m3 b/m3-sys/m3front/src/misc/M3Header.m3 >> index 1e4decf..877d77c 100644 >> --- a/m3-sys/m3front/src/misc/M3Header.m3 >> +++ b/m3-sys/m3front/src/misc/M3Header.m3 >> @@ -104,7 +104,7 @@ PROCEDURE PushGeneric (VAR s: State) = >> IF (s.generic = NIL) THEN s.failed := TRUE; RETURN; END; >> >> (* build a synthetic file name & start reading *) >> - filename := old_filename & " => " & filename; >> + filename := Host.FileTail(old_filename) & " => " & filename; >> Scanner.Push (filename, s.generic, is_main := Scanner.in_main); >> >> (* make sure we got what we wanted *) >> diff --git a/m3-sys/m3front/src/misc/Scanner.m3 b/m3-sys/m3front/src/misc/Scanner.m3 >> index 7470374..e1dc024 100644 >> --- a/m3-sys/m3front/src/misc/Scanner.m3 >> +++ b/m3-sys/m3front/src/misc/Scanner.m3 >> @@ -228,13 +228,16 @@ PROCEDURE Here (VAR file: TEXT; VAR line: INTEGER) = >> BEGIN >> file := files [offset DIV MaxLines]; >> line := offset MOD MaxLines; >> + IF Target.ReduceTargetVariation THEN >> + file := Host.FileTail(file); >> + END; >> END Here; >> >> PROCEDURE LocalHere (VAR file: TEXT; VAR line: INTEGER) = >> VAR fnum := offset DIV MaxLines; >> BEGIN >> IF (local_files[fnum] = NIL) THEN >> - local_files[fnum] := files[fnum]; >> + local_files[fnum] := Host.FileTail(files[fnum]); >> END; >> file := local_files [fnum]; >> line := offset MOD MaxLines; >> diff --git a/m3-sys/m3front/src/values/Module.m3 b/m3-sys/m3front/src/values/Module.m3 >> index a085eab..576c857 100644 >> --- a/m3-sys/m3front/src/values/Module.m3 >> +++ b/m3-sys/m3front/src/values/Module.m3 >> @@ -421,7 +421,7 @@ PROCEDURE PushGeneric (t: T; VAR rd: File.T): M3ID.T = >> END; >> >> (* build a synthetic file name & start reading *) >> - filename := old_filename & " => " & filename; >> + filename := Host.FileTail(old_filename) & " => " & filename; >> Scanner.Push (filename, rd, is_main := Scanner.in_main); >> t.genericFile := filename; >> >> diff --git a/m3-sys/m3middle/src/Target.i3 b/m3-sys/m3middle/src/Target.i3 >> index fe198d2..cd7acee 100644 >> --- a/m3-sys/m3middle/src/Target.i3 >> +++ b/m3-sys/m3middle/src/Target.i3 >> @@ -529,4 +529,8 @@ VAR (*CONST*) >> test for nested procedures passed as parameters must be more >> elaborate (e.g. HPPA). *) >> >> + (* This removes some unnecessary target variation in the output, >> + * such as current working directory in debug output. *) >> + ReduceTargetVariation: BOOLEAN; >> + >> END Target. >> diff --git a/scripts/python/pylib.py b/scripts/python/pylib.py >> index 73e622f..487ad17 100755 >> --- a/scripts/python/pylib.py >> +++ b/scripts/python/pylib.py >> @@ -388,7 +388,7 @@ def _GetAllTargets(): >> >> _CBackend = "c" in sys.argv or "C" in sys.argv >> _BuildDirC = ["", "c"][_CBackend] >> -_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why"] >> +_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why", "reduce-target-variation", "reducetargetvariation"] >> _SkipGccFlags = ["nogcc", "skipgcc", "omitgcc"] >> _PossiblePylibFlags = ["noclean", "nocleangcc", "c", "C"] + _SkipGccFlags + _PossibleCm3Flags >> >> >> Thank you, >> - Jay >> >> >> _______________________________________________ >> M3devel mailing list >> M3devel at elegosoft.com >> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> > > -- > Rodney Bates > rodney.m.bates at acm.org > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel From jay.krell at cornell.edu Mon Jul 4 23:17:23 2016 From: jay.krell at cornell.edu (Jay K) Date: Mon, 4 Jul 2016 21:17:23 +0000 Subject: [M3devel] defaulting to parallel backend? In-Reply-To: <577A78EA.5000600@lcwb.coop> References: , <577A78EA.5000600@lcwb.coop> Message-ID: ok. Presumably something like: name it to match C++11 They call it: unsigned std::thread::hardware_concurrency(void); We should call it: PROCEDURE Thread.HardwareConcurrency(): INTEGER; OR PROCEDURE Thread.HardwareConcurrency(): Word.T; One could specify the following result for "unknown": <0, 0, 1 1 is nice and safe, but lossy vs. actually knowing it is 1. One could add another function for the known-ness boolean: PROCEDURE Thread.HardwareConcurrencyValid(): BOOLEAN; or PROCEDURE Thread.HardwareConcurrencyKnown(): BOOLEAN; "Unknown" I don't believe would really stem from the underlying implementation, there is no error path in the underlying operating systems, but be for platforms we don't have a port for, so is less interesting and just returning a sub-par 1 isn't so bad. ?- Jay ---------------------------------------- > Date: Mon, 4 Jul 2016 09:55:38 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com > Subject: Re: [M3devel] defaulting to parallel backend? > > I prefer to use one fewer than the number of processors, so I can > web browse, read email, review code, etc. while waiting for a > compile. The compiler can saturate them all, making any other > stuff very slow. > > On 07/04/2016 03:45 AM, Jay K wrote: >> I suggest we make parallel backend the default. >> >> You can turn it off with -pb 1. >> >> Turn it "up" with -pb >> >> And the default is a thread per processor. >> This will be kernel specific code, but we can get far with >> >> 1) Win32 GetSystemInfo, if applicable (e.g. C backend) >> 2) sysconf(_SC_NPROCESSORS_ONLN) >> >> >> and I'll research the various OSes (Darwin, Linux, *BSD, Solaris, and possibly AIX, HP-UX, Irix, Mingw, Cygwin). >> ok? >> >> I could really use the speed boost and multi-processor machines are overwhelmingly common now (though easily avoided with VMs). >> >> >> - Jay >> >> >> _______________________________________________ >> M3devel mailing list >> M3devel at elegosoft.com >> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> > > -- > Rodney Bates > rodney.m.bates at acm.org > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel From jay.krell at cornell.edu Tue Jul 5 09:54:14 2016 From: jay.krell at cornell.edu (Jay K) Date: Tue, 5 Jul 2016 07:54:14 +0000 Subject: [M3devel] some code from boost? Message-ID: I'd like to bring in some code from boost. To m3core -- Thread.HardwareConcurrency, i.e. the number of processors It is licensed like so: Boost Software License - Version 1.0 - August 17th, 2003 Permission is hereby granted, free of charge, to any person or organization obtaining a copy of the software and accompanying documentation covered by this license (the "Software") to use, reproduce, display, distribute, execute, and transmit the Software, and to prepare derivative works of the Software, and to permit third-parties to whom the Software is furnished to do so, all subject to the following: The copyright notices in the Software and this entire statement, including the above license grant, this restriction and the following disclaimer, must be included in all copies of the Software, in whole or in part, and all derivative works of the Software, unless such copies or derivative works are solely in the form of machine-executable object code generated by a source language processor. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. http://www.boost.org/LICENSE_1_0.txt) I find the license a little long and would rather not have more, but it is a good body of work overall. I would put the code in circa m3core/thread/boost/win32, m3core/thread/boost/posix or maybe m3core/thread/Common/ThreadBoostWin32.c ThreadBoostPosix.c ok? esp. on the licensing? It is just a few lines of code. They call this thread::hardware_concurrency and it is the number of CPUs or cores (this number is actually ambiguous these days, with packages, cores, and hyper-threading). They have another number called physical_concurrency we might want to also provide. They have implementations for at least the following the systems: ?Win32 GetSystemInfo ?glibc?get_nprocs ?sysconf(_SC_NPROCESSORS_ONLN) ?Apple and FreeBSD?sysctlbyname("hw.ncpu") ?HP-UX?pthread_num_processors_np For licensing or at least credit purposs, it should be two files -- Win32 and Posix. The code is C++ but can be changed to C, and for should should be. I suppose, if the license is too restrictive, I can keep it out of m3core and put it only in cm3? ?- Jay From jay.krell at cornell.edu Tue Jul 5 10:19:23 2016 From: jay.krell at cornell.edu (Jay K) Date: Tue, 5 Jul 2016 08:19:23 +0000 Subject: [M3devel] purported condition variable problems on Win32? Message-ID: https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV vs. http://www.cs.wustl.edu/~schmidt/win32-cv-1.html vs. https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain I wrote this: PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; alertable: BOOLEAN) RAISES {Alerted} = (* LL = m on entry and exit, but not for the duration * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp * NOTE that they merge the user lock and the condition lock. * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html * "3.3. The Generation Count Solution" *) Do we have the problems described in README.CV? I haven't looked through the ACE code to see to what extent they resemble solution 3.3, or if they changed as a result of this discussion -- which I admit I don't understandand haven't read closely. Spurious wakeups are ok, though should be minimized. I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. Thank you, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Jul 5 10:26:36 2016 From: jay.krell at cornell.edu (Jay K) Date: Tue, 5 Jul 2016 08:26:36 +0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: Message-ID: Here is another implementation to consider: https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Tue, 5 Jul 2016 08:19:23 +0000 Subject: [M3devel] purported condition variable problems on Win32? https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV vs. http://www.cs.wustl.edu/~schmidt/win32-cv-1.html vs. https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain I wrote this: PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; alertable: BOOLEAN) RAISES {Alerted} = (* LL = m on entry and exit, but not for the duration * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp * NOTE that they merge the user lock and the condition lock. * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html * "3.3. The Generation Count Solution" *) Do we have the problems described in README.CV? I haven't looked through the ACE code to see to what extent they resemble solution 3.3, or if they changed as a result of this discussion -- which I admit I don't understandand haven't read closely. Spurious wakeups are ok, though should be minimized. I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. Thank you, - Jay _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From lemming at henning-thielemann.de Tue Jul 5 18:34:54 2016 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Tue, 5 Jul 2016 18:34:54 +0200 (CEST) Subject: [M3devel] defaulting to parallel backend? In-Reply-To: References: Message-ID: On Mon, 4 Jul 2016, Jay K wrote: > I suggest we make parallel backend the default. Generally I do not like if a program eats up all computing power without being told so. E.g. my machine has 8 hyperthreads but only 4 cores. Running on 8 processes hardly gives maximum efficiency, most oftenly speed is maximal at 3 processes. I also like the idea to use the remaining computing power for different tasks. Please just leave the choice of the number of processes to the user. From lemming at henning-thielemann.de Tue Jul 5 18:36:45 2016 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Tue, 5 Jul 2016 18:36:45 +0200 (CEST) Subject: [M3devel] defaulting to parallel backend? In-Reply-To: <577A78EA.5000600@lcwb.coop> References: <577A78EA.5000600@lcwb.coop> Message-ID: On Mon, 4 Jul 2016, Rodney M. Bates wrote: > I prefer to use one fewer than the number of processors, so I can > web browse, read email, review code, etc. while waiting for a > compile. The compiler can saturate them all, making any other > stuff very slow. Slowdown can happen even earlier if the compilation is bound by disk access time. From jay.krell at cornell.edu Wed Jul 6 07:44:45 2016 From: jay.krell at cornell.edu (Jay K) Date: Wed, 6 Jul 2016 05:44:45 +0000 Subject: [M3devel] defaulting to parallel backend? In-Reply-To: References: , <577A78EA.5000600@lcwb.coop>, Message-ID: There is no good easy answer here. You really want everything on the computer to use all of the available resources, to service whatever the human wants done as fast as possible, while remaining responsive e.g. to requests to cancel or requests to increase the work load. This is largely the job of the kernel, but it is an impossible job. To some extent we are obligated to over subscribe, in order to provide the kernel more information. But, again, it is an impossible job. Besides it probably being NP complete to complete the submitted work as soon as possible, we also don't know the user's priorities. I could propose "pb 1" to mean run single threaded, putting the burden on you to give. But that is a little rude. Note that historically you didn't have as much flexibility here -- single core systems. Anyway, how about the default be half? Or 1 if <= 3 cores available? I seem to have 4 cores, so this will help me. I understand this is still not great, e.g. if you have 10 high priority single processors jobs, and one low priority Modula-3 build, Modula-3 will by default use more than you wish. And yes, I understand disk are another resource. And network -- disk can be on network. You want to operate at the "edge of resource exhaustion" in general. - Jay > Date: Tue, 5 Jul 2016 18:36:45 +0200 > From: lemming at henning-thielemann.de > To: rodney.m.bates at acm.org > Subject: Re: [M3devel] defaulting to parallel backend? > CC: m3devel at elegosoft.com > > > On Mon, 4 Jul 2016, Rodney M. Bates wrote: > > > I prefer to use one fewer than the number of processors, so I can > > web browse, read email, review code, etc. while waiting for a > > compile. The compiler can saturate them all, making any other > > stuff very slow. > > Slowdown can happen even earlier if the compilation is bound by disk > access time. > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Jul 7 11:24:08 2016 From: jay.krell at cornell.edu (Jay K) Date: Thu, 7 Jul 2016 09:24:08 +0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: , Message-ID: So...I do NOT understand all of this. However, comparing the three implementationsand attempting to understand ours,I am struck by the following - Our broadcast seems ok, but - our signal seems to wake everyone, and then...they race a bit, one will decide it is last, and reset the event, but every prior waiter is still woken. Why not merge the following chunks: WHILE (NOT alerted) AND (NOT waitDone) DO... 1 EnterCriticalSection(conditionLock); waitDone := (c.tickets # 0 AND c.counter # count); LeaveCriticalSection(conditionLock); END; (* WHILE *) IF waitDone THEN alerted := FALSE; END; m.acquire(); 2 EnterCriticalSection(conditionLock); DEC(c.waiters); IF waitDone THEN esp. here. DEC(c.tickets); lastWaiter := (c.tickets = 0); END; LeaveCriticalSection(conditionLock); That is, if we decrement tickets earlier within the first critical section, while we will still wake everyone, only one will decide waitDone and the rest will keep looping. A downside of this, perhaps, is that waking all for Broadcast might be a little slower. but more so, in both the "ptw32" (pthreads for win32) and Boost threads implementations, they seem to deal with this differently than us, and they each do about the same thing -- they use a counted semaphore. In the boost case, it appears they duplicate the semaphore for every notification generation, which seems expensive. Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. Or they can do their own reference counting? jdk7 seems to looke like jdk6. The code is gone in jdk8 and I can't easily find the delete in history. The lock merging jdk does probably helps here too. In fact they merge #1 and #2 above as a result. But they still initially wake all threads for signal, not just broadcast. There is also the problem that all the event waits are followed by EnterCriticalSection (or jdk facsimile). - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: RE: [M3devel] purported condition variable problems on Win32? Date: Tue, 5 Jul 2016 08:26:36 +0000 Here is another implementation to consider: https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Tue, 5 Jul 2016 08:19:23 +0000 Subject: [M3devel] purported condition variable problems on Win32? https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV vs. http://www.cs.wustl.edu/~schmidt/win32-cv-1.html vs. https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain I wrote this: PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; alertable: BOOLEAN) RAISES {Alerted} = (* LL = m on entry and exit, but not for the duration * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp * NOTE that they merge the user lock and the condition lock. * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html * "3.3. The Generation Count Solution" *) Do we have the problems described in README.CV? I haven't looked through the ACE code to see to what extent they resemble solution 3.3, or if they changed as a result of this discussion -- which I admit I don't understandand haven't read closely. Spurious wakeups are ok, though should be minimized. I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. Thank you, - Jay _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Wed Jul 13 20:26:59 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Wed, 13 Jul 2016 13:26:59 -0500 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: Message-ID: <578687F3.1010108@lcwb.coop> I've begun looking at this. On 07/05/2016 03:19 AM, Jay K wrote: > https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > As for spurious wakeups, the M3 Thread.i3 documentation of Signal explicitly allows this: "One or more threads waiting on c become eligible to run", whereas the posix pthread_cond_signal, according to schmidt, "notifies one thread waiting on condition variable cv". As I recall, these (the M3 semantics) agree with the original Brinch-Hansen condition variables. So, for implementing posix CVs, this would be a true bug, but not so for M3 Thread CVs. > vs. > > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > vs. > https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > I wrote this: > PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > alertable: BOOLEAN) RAISES {Alerted} = > (* LL = m on entry and exit, but not for the duration > * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > * NOTE that they merge the user lock and the condition lock. > * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > * "3.3. The Generation Count Solution" > *) > > Do we have the problems described in README.CV? > > I haven't looked through the ACE code to see > to what extent they resemble solution 3.3, or if they > changed as a result of this discussion -- which I admit I don't understand > and haven't read closely. > > > Spurious wakeups are ok, though should be minimized. > > I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > Thank you, > - Jay > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Sun Jul 17 23:06:19 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 17 Jul 2016 21:06:19 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: , Message-ID: <578BF33B.2030605@lcwb.coop> I spent some time looking over the signal part of ThreadWin32.m3. A few comments: This looks like a very direct implementation of Schmidts' generation count algorithm, with a number of important little details additionally taken care of. On small difference is at line 308 (B, below) where Schmidt does c.counter>count instead of #. Usually, this would make no difference, since counter only increases, but on a long-running program, it could overflow. If this happened,as it often does, as silent wrap-around to FIRST(INTEGER), some threads could be trapped for an extremely long time, waiting for counter come back around greater than their count. As coded in ThreadWin32, they would be allowed to proceed normally. The only glitch would be that, if there were simultaneously waiting threads separated by NUMBER(INTEGER) truly different generations (but the same value of c.counter), the recent ones could unfairly proceed along with the extremely old ones. That seems extremely less likely than the mere existence of an overflow. Both type Condition and type Activation have a field waitEvent: HANDLE. Condition.waitEvent is extensively used, but Activation.waitEvent is only created at :537 and deleted at :554, but never used in any real way. It could be deleted. As for this code: EnterCriticalSection(conditionLock); (* Capture the value of the counter before we start waiting. * We will not stop waiting until the counter changes. * That is, we will not stop waiting until a signal * comes in after we start waiting. *) count := c.counter; INC(c.waiters); LeaveCriticalSection(conditionLock); m.release(); (* can this be moved to before taking conditionLock? *) No, it is important to sample and save c.counter in the order threads wait. Retaining m until this is done ensures this. Otherwise, some other thread that waited later could have altered c.counter before this one gets conditionLock and saves its copy of c.counter. I have made several comment changes to ThreadWin32.m3 that would have made it easier to vet. I will commit these, but only comments. It is either very difficult or impossible for me to test or even recompile this module, and, especially as fragile as this kind of code is, I don't want to commit any substantive changes untested and uncompiled. More below: On 07/07/2016 04:24 AM, Jay K wrote: > So...I do NOT understand all of this. > > > However, comparing the three implementations > and attempting to understand ours, > I am struck by the following > > - Our broadcast seems ok, but > - our signal seems to wake everyone, > and then...they race a bit, > one will decide it is last, and > reset the event, but every prior waiter > is still woken. > Why not merge the following chunks: I don't think this will work. It is important that, once a thread has decided, at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, if it is also the last of its generation to proceed, resets c.waitevent. This is ensured by its having already reacquired m first, preventing any other thread from returning from its Wait yet. And it can't attempt to acquire m while already holding conditionLock, because this would invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while already holding m, at line 265. > > > WHILE (NOT alerted) AND (NOT waitDone) DO > . > . > . > 1 > EnterCriticalSection(conditionLock); > waitDone := (c.tickets # 0 AND c.counter # count); A:---------^ B:--------------------------------------------------^ > LeaveCriticalSection(conditionLock); > END; (* WHILE *) > IF waitDone THEN > alerted := FALSE; > END; > m.acquire(); > 2 > EnterCriticalSection(conditionLock); > DEC(c.waiters); > IF waitDone THEN esp. here. > DEC(c.tickets); C:---------------^ > lastWaiter := (c.tickets = 0); > END; > LeaveCriticalSection(conditionLock); > > > That is, if we decrement tickets earlier > within the first critical section, > while we will still wake everyone, > only one will decide waitDone and the rest will keep looping. > A downside of this, perhaps, is that waking all for Broadcast > might be a little slower. > but more so, in both the "ptw32" (pthreads for win32) and Boost threads > implementations, they seem to deal with this differently than us, > and they each do about the same thing -- they use a counted semaphore. > In the boost case, it appears they duplicate the semaphore for every > notification generation, which seems expensive. > > Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > Or they can do their own reference counting? > jdk7 seems to looke like jdk6. > The code is gone in jdk8 and I can't easily find the delete in history. > The lock merging jdk does probably helps here too. > In fact they merge #1 and #2 above as a result. > But they still initially wake all threads for signal, not just broadcast. > There is also the problem that all the event waits > are followed by EnterCriticalSection (or jdk facsimile). > - Jay > > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Subject: RE: [M3devel] purported condition variable problems on Win32? > Date: Tue, 5 Jul 2016 08:26:36 +0000 > > Here is another implementation to consider: > > https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > > - Jay > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Tue, 5 Jul 2016 08:19:23 +0000 > Subject: [M3devel] purported condition variable problems on Win32? > > https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > > vs. > > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > vs. > https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > I wrote this: > PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > alertable: BOOLEAN) RAISES {Alerted} = > (* LL = m on entry and exit, but not for the duration > * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > * NOTE that they merge the user lock and the condition lock. > * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > * "3.3. The Generation Count Solution" > *) > > Do we have the problems described in README.CV? > > I haven't looked through the ACE code to see > to what extent they resemble solution 3.3, or if they > changed as a result of this discussion -- which I admit I don't understand > and haven't read closely. > > > Spurious wakeups are ok, though should be minimized. > > I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > Thank you, > - Jay > > _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Sun Jul 17 23:23:50 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 17 Jul 2016 21:23:50 -0000 Subject: [M3devel] defaulting to parallel backend? In-Reply-To: References: , <577A78EA.5000600@lcwb.coop>, Message-ID: <578BF74F.1020009@lcwb.coop> For systems that have it, how about just using the 'nice' command? On 07/06/2016 12:44 AM, Jay K wrote: > There is no good easy answer here. > > > You really want everything on the computer to use all of the > available resources, to service whatever the human wants done > as fast as possible, while remaining responsive e.g. to > requests to cancel or requests to increase the work load. > > > This is largely the job of the kernel, but it is an impossible job. > > To some extent we are obligated to over subscribe, in order > to provide the kernel more information. > But, again, it is an impossible job. > > Besides it probably being NP complete to complete the submitted work > as soon as possible, we also don't know the user's priorities. > > I could propose "pb 1" to mean run single threaded, putting > the burden on you to give. But that is a little rude. > > Note that historically you didn't have as much flexibility here -- single core systems. > > > Anyway, how about the default be half? > Or 1 if <= 3 cores available? > I seem to have 4 cores, so this will help me. > > > I understand this is still not great, e.g. if you have 10 > high priority single processors jobs, and one low priority > Modula-3 build, Modula-3 will by default use more than you wish. > > And yes, I understand disk are another resource. > And network -- disk can be on network. > You want to operate at the "edge of resource exhaustion" in general. > > > - Jay > > > > > > Date: Tue, 5 Jul 2016 18:36:45 +0200 > > From: lemming at henning-thielemann.de > > To: rodney.m.bates at acm.org > > Subject: Re: [M3devel] defaulting to parallel backend? > > CC: m3devel at elegosoft.com > > > > > > On Mon, 4 Jul 2016, Rodney M. Bates wrote: > > > > > I prefer to use one fewer than the number of processors, so I can > > > web browse, read email, review code, etc. while waiting for a > > > compile. The compiler can saturate them all, making any other > > > stuff very slow. > > > > Slowdown can happen even earlier if the compilation is bound by disk > > access time. > > _______________________________________________ > > M3devel mailing list > > M3devel at elegosoft.com > > https://m3lists.elegosoft.com/mailman/listinfo/m3devel -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Mon Jul 18 16:26:28 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 18 Jul 2016 14:26:28 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: <578BF33B.2030605@lcwb.coop> References: <578BF33B.2030605@lcwb.coop> Message-ID: <578CE6C7.2010007@lcwb.coop> I spent some time looking over the signal part of ThreadWin32.m3. A few comments: This looks like a very direct implementation of Schmidts' generation count algorithm, with a number of important little details additionally taken care of. On small difference is at line 308 (B, below) where Schmidt does c.counter>count instead of #. Usually, this would make no difference, since counter only increases, but on a long-running program, it could overflow. If this happened,as it often does, as silent wrap-around to FIRST(INTEGER), some threads could be trapped for an extremely long time, waiting for counter come back around greater than their count. As coded in ThreadWin32, they would be allowed to proceed normally. The only glitch would be that, if there were simultaneously waiting threads separated by NUMBER(INTEGER) truly different generations (but the same value of c.counter), the recent ones could unfairly proceed along with the extremely old ones. That seems extremely less likely than the mere existence of an overflow. Both type Condition and type Activation have a field waitEvent: HANDLE. Condition.waitEvent is extensively used, but Activation.waitEvent is only created at :537 and deleted at :554, but never used in any real way. It could be deleted. As for this code: EnterCriticalSection(conditionLock); (* Capture the value of the counter before we start waiting. * We will not stop waiting until the counter changes. * That is, we will not stop waiting until a signal * comes in after we start waiting. *) count := c.counter; INC(c.waiters); LeaveCriticalSection(conditionLock); m.release(); (* can this be moved to before taking conditionLock? *) No, it is important to sample and save c.counter in the order threads wait. Retaining m until this is done ensures this. Otherwise, some other thread that waited later could have altered c.counter before this one gets conditionLock and saves its copy of c.counter. I have made several comment changes to ThreadWin32.m3 that would have made it easier to vet. I will commit these, but only comments. It is either very difficult or impossible for me to test or even recompile this module, and, especially as fragile as this kind of code is, I don't want to commit any substantive changes untested and uncompiled. More below: On 07/07/2016 04:24 AM, Jay K wrote: > So...I do NOT understand all of this. > > > However, comparing the three implementations > and attempting to understand ours, > I am struck by the following > > - Our broadcast seems ok, but > - our signal seems to wake everyone, > and then...they race a bit, > one will decide it is last, and > reset the event, but every prior waiter > is still woken. > Why not merge the following chunks: I don't think this will work. It is important that, once a thread has decided, at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, if it is also the last of its generation to proceed, resets c.waitevent. This is ensured by its having already reacquired m first, preventing any other thread from returning from its Wait yet. And it can't attempt to acquire m while already holding conditionLock, because this would invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while already holding m, at line 265. > > > WHILE (NOT alerted) AND (NOT waitDone) DO > . > . > . > 1 > EnterCriticalSection(conditionLock); > waitDone := (c.tickets # 0 AND c.counter # count); A:---------^ B:--------------------------------------------------^ > LeaveCriticalSection(conditionLock); > END; (* WHILE *) > IF waitDone THEN > alerted := FALSE; > END; > m.acquire(); > 2 > EnterCriticalSection(conditionLock); > DEC(c.waiters); > IF waitDone THEN esp. here. > DEC(c.tickets); C:---------------^ > lastWaiter := (c.tickets = 0); > END; > LeaveCriticalSection(conditionLock); > > > That is, if we decrement tickets earlier > within the first critical section, > while we will still wake everyone, > only one will decide waitDone and the rest will keep looping. > A downside of this, perhaps, is that waking all for Broadcast > might be a little slower. > but more so, in both the "ptw32" (pthreads for win32) and Boost threads > implementations, they seem to deal with this differently than us, > and they each do about the same thing -- they use a counted semaphore. > In the boost case, it appears they duplicate the semaphore for every > notification generation, which seems expensive. > > Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > Or they can do their own reference counting? > jdk7 seems to looke like jdk6. > The code is gone in jdk8 and I can't easily find the delete in history. > The lock merging jdk does probably helps here too. > In fact they merge #1 and #2 above as a result. > But they still initially wake all threads for signal, not just broadcast. > There is also the problem that all the event waits > are followed by EnterCriticalSection (or jdk facsimile). > - Jay > > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Subject: RE: [M3devel] purported condition variable problems on Win32? > Date: Tue, 5 Jul 2016 08:26:36 +0000 > > Here is another implementation to consider: > > https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > > - Jay > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Tue, 5 Jul 2016 08:19:23 +0000 > Subject: [M3devel] purported condition variable problems on Win32? > > https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > > vs. > > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > vs. > https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > I wrote this: > PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > alertable: BOOLEAN) RAISES {Alerted} = > (* LL = m on entry and exit, but not for the duration > * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > * NOTE that they merge the user lock and the condition lock. > * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > * "3.3. The Generation Count Solution" > *) > > Do we have the problems described in README.CV? > > I haven't looked through the ACE code to see > to what extent they resemble solution 3.3, or if they > changed as a result of this discussion -- which I admit I don't understand > and haven't read closely. > > > Spurious wakeups are ok, though should be minimized. > > I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > Thank you, > - Jay > > _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Tue Jul 19 03:53:24 2016 From: jay.krell at cornell.edu (Jay) Date: Tue, 19 Jul 2016 01:53:24 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: <578CE6C7.2010007@lcwb.coop> References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop> Message-ID: <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. Otherwise previous version I think had a very large lock. Changing the < to !=, is that really ok? I get that 32bit overflow is possible. We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. I guess you are saying it would be unfair, which is ok. There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. I'll check on the unused event. - Jay > On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > > > > > > I spent some time looking over the signal part of ThreadWin32.m3. A few > comments: > > This looks like a very direct implementation of Schmidts' generation count > algorithm, with a number of important little details additionally taken care > of. > > On small difference is at line 308 (B, below) where Schmidt does > c.counter>count instead of #. Usually, this would make no difference, > since counter only increases, but on a long-running program, it could > overflow. If this happened,as it often does, as silent wrap-around to > FIRST(INTEGER), some threads could be trapped for an extremely long > time, waiting for counter come back around greater than their count. As > coded in ThreadWin32, they would be allowed to proceed normally. > > The only glitch would be that, if there were simultaneously waiting threads > separated by NUMBER(INTEGER) truly different generations (but the same value > of c.counter), the recent ones could unfairly proceed along with the > extremely old ones. That seems extremely less likely than the > mere existence of an overflow. > > Both type Condition and type Activation have a field waitEvent: HANDLE. > Condition.waitEvent is extensively used, but Activation.waitEvent is > only created at :537 and deleted at :554, but never used in any real > way. It could be deleted. > > As for this code: > > EnterCriticalSection(conditionLock); > > (* Capture the value of the counter before we start waiting. > * We will not stop waiting until the counter changes. > * That is, we will not stop waiting until a signal > * comes in after we start waiting. > *) > > count := c.counter; > INC(c.waiters); > > LeaveCriticalSection(conditionLock); > m.release(); (* can this be moved to before taking conditionLock? *) > > No, it is important to sample and save c.counter in the order threads wait. > Retaining m until this is done ensures this. Otherwise, some other thread > that waited later could have altered c.counter before this one gets conditionLock > and saves its copy of c.counter. > > I have made several comment changes to ThreadWin32.m3 that would have made > it easier to vet. I will commit these, but only comments. It is either > very difficult or impossible for me to test or even recompile this module, > and, especially as fragile as this kind of code is, I don't want to commit > any substantive changes untested and uncompiled. > > More below: > >> On 07/07/2016 04:24 AM, Jay K wrote: >> So...I do NOT understand all of this. >> >> >> However, comparing the three implementations >> and attempting to understand ours, >> I am struck by the following >> >> - Our broadcast seems ok, but >> - our signal seems to wake everyone, >> and then...they race a bit, >> one will decide it is last, and >> reset the event, but every prior waiter >> is still woken. >> Why not merge the following chunks: > > I don't think this will work. It is important that, once a thread has decided, > at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > if it is also the last of its generation to proceed, resets c.waitevent. This > is ensured by its having already reacquired m first, preventing any other thread > from returning from its Wait yet. > > And it can't attempt to acquire m while already holding conditionLock, because this would > invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > already holding m, at line 265. > >> >> >> WHILE (NOT alerted) AND (NOT waitDone) DO >> . >> . >> . >> 1 >> EnterCriticalSection(conditionLock); >> waitDone := (c.tickets # 0 AND c.counter # count); > A:---------^ > B:--------------------------------------------------^ >> LeaveCriticalSection(conditionLock); >> END; (* WHILE *) >> IF waitDone THEN >> alerted := FALSE; >> END; >> m.acquire(); >> 2 >> EnterCriticalSection(conditionLock); >> DEC(c.waiters); >> IF waitDone THEN esp. here. >> DEC(c.tickets); > > C:---------------^ >> lastWaiter := (c.tickets = 0); >> END; >> LeaveCriticalSection(conditionLock); >> >> >> That is, if we decrement tickets earlier >> within the first critical section, >> while we will still wake everyone, >> only one will decide waitDone and the rest will keep looping. >> A downside of this, perhaps, is that waking all for Broadcast >> might be a little slower. >> but more so, in both the "ptw32" (pthreads for win32) and Boost threads >> implementations, they seem to deal with this differently than us, >> and they each do about the same thing -- they use a counted semaphore. >> In the boost case, it appears they duplicate the semaphore for every >> notification generation, which seems expensive. >> >> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. >> Or they can do their own reference counting? >> jdk7 seems to looke like jdk6. >> The code is gone in jdk8 and I can't easily find the delete in history. >> The lock merging jdk does probably helps here too. >> In fact they merge #1 and #2 above as a result. >> But they still initially wake all threads for signal, not just broadcast. >> There is also the problem that all the event waits >> are followed by EnterCriticalSection (or jdk facsimile). >> - Jay >> >> >> >> >> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --- >> From: jay.krell at cornell.edu >> To: m3devel at elegosoft.com >> Subject: RE: [M3devel] purported condition variable problems on Win32? >> Date: Tue, 5 Jul 2016 08:26:36 +0000 >> >> Here is another implementation to consider: >> >> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp >> >> - Jay >> >> >> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --- >> From: jay.krell at cornell.edu >> To: m3devel at elegosoft.com >> Date: Tue, 5 Jul 2016 08:19:23 +0000 >> Subject: [M3devel] purported condition variable problems on Win32? >> >> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV >> >> vs. >> >> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >> vs. >> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain >> I wrote this: >> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; >> alertable: BOOLEAN) RAISES {Alerted} = >> (* LL = m on entry and exit, but not for the duration >> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp >> * NOTE that they merge the user lock and the condition lock. >> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >> * "3.3. The Generation Count Solution" >> *) >> >> Do we have the problems described in README.CV? >> >> I haven't looked through the ACE code to see >> to what extent they resemble solution 3.3, or if they >> changed as a result of this discussion -- which I admit I don't understand >> and haven't read closely. >> >> >> Spurious wakeups are ok, though should be minimized. >> >> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. >> >> Thank you, >> - Jay >> >> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> >> >> _______________________________________________ >> M3devel mailing list >> M3devel at elegosoft.com >> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > -- > Rodney Bates > rodney.m.bates at acm.org > > From rodney_bates at lcwb.coop Wed Jul 20 18:44:24 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Wed, 20 Jul 2016 16:44:24 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop> <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> Message-ID: <578FAA2F.6020105@lcwb.coop> In looking at ThreadWin32.m3 more, I think I see a significant bug. This gets confusing, since there are Modula-3 waits, signals, etc. and similar concepts in Win32, but which are not the same. I am going the try to put "M3-" and "Win-" prefixes on things to make this clearer. I cartainly need to do this for my own benefit, if for nobody else. Suppose two threads are both M3-waiting in Wait(m,c,...), further Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. c.tickets = 0, and c.waiters = 2. Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), which will allow both the waiters to Win-wakeup sometime soon. Before releasing c.lock, Signal increments c.tickets to 1 (supposedly to eventually allow only one waiter to M3-wakeup) and increments c.counter, setting things up so that the waitDone expression at :308 will be TRUE for the waiting threads, the next time they evaluate it. Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, releases c.lock, and tries to acquire m. The problem arises because the second waiter can get to these steps before the first can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The second to reach :308 will find c.tickets # 0, as did the first, and proceed. Each will eventually, do a M3-wake. If this were Posix condition variables, this would be wrong, because Posix says only one waiter proceeds. For M3, it's OK, if unnecessary, because M3 says one or more waiters proceed. However, each will, in the process, get to :322, and DEC(c.tickets). Signal only provided one ticket, but the waiters have taken two. c.tickets goes negative. After this, when some thread can do another (M3-)Wait, and some other does another (M3-)Signal, c.tickets increments only back to zero, which means the new waiter will not M3-wake, even though it should. I am going to try to construct a test case that will force this scenario. See more below: On 07/18/2016 08:53 PM, Jay wrote: > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > Otherwise previous version I think had a very large lock. > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > I guess you are saying it would be unfair, which is ok. > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > I'll check on the unused event. > > - Jay > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: >> >> >> >> >> >> I spent some time looking over the signal part of ThreadWin32.m3. A few >> comments: >> >> This looks like a very direct implementation of Schmidts' generation count >> algorithm, with a number of important little details additionally taken care >> of. >> >> On small difference is at line 308 (B, below) where Schmidt does >> c.counter>count instead of #. Usually, this would make no difference, >> since counter only increases, but on a long-running program, it could >> overflow. If this happened,as it often does, as silent wrap-around to >> FIRST(INTEGER), some threads could be trapped for an extremely long >> time, waiting for counter come back around greater than their count. As >> coded in ThreadWin32, they would be allowed to proceed normally. >> >> The only glitch would be that, if there were simultaneously waiting threads >> separated by NUMBER(INTEGER) truly different generations (but the same value >> of c.counter), the recent ones could unfairly proceed along with the >> extremely old ones. That seems extremely less likely than the >> mere existence of an overflow. >> >> Both type Condition and type Activation have a field waitEvent: HANDLE. >> Condition.waitEvent is extensively used, but Activation.waitEvent is >> only created at :537 and deleted at :554, but never used in any real >> way. It could be deleted. >> >> As for this code: >> >> EnterCriticalSection(conditionLock); >> >> (* Capture the value of the counter before we start waiting. >> * We will not stop waiting until the counter changes. >> * That is, we will not stop waiting until a signal >> * comes in after we start waiting. >> *) >> >> count := c.counter; >> INC(c.waiters); >> >> LeaveCriticalSection(conditionLock); >> m.release(); (* can this be moved to before taking conditionLock? *) >> >> No, it is important to sample and save c.counter in the order threads wait. >> Retaining m until this is done ensures this. Otherwise, some other thread >> that waited later could have altered c.counter before this one gets conditionLock >> and saves its copy of c.counter. >> >> I have made several comment changes to ThreadWin32.m3 that would have made >> it easier to vet. I will commit these, but only comments. It is either >> very difficult or impossible for me to test or even recompile this module, >> and, especially as fragile as this kind of code is, I don't want to commit >> any substantive changes untested and uncompiled. >> >> More below: >> >>> On 07/07/2016 04:24 AM, Jay K wrote: >>> So...I do NOT understand all of this. >>> >>> >>> However, comparing the three implementations >>> and attempting to understand ours, >>> I am struck by the following >>> >>> - Our broadcast seems ok, but >>> - our signal seems to wake everyone, >>> and then...they race a bit, >>> one will decide it is last, and >>> reset the event, but every prior waiter >>> is still woken. >>> Why not merge the following chunks: >> >> I don't think this will work. It is important that, once a thread has decided, >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, >> if it is also the last of its generation to proceed, resets c.waitevent. This >> is ensured by its having already reacquired m first, preventing any other thread >> from returning from its Wait yet. I was wrong about this. There is already no fairness being enforced here. Even with only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is arbitrary anyway. There is no benefit in trying to control the order they reacquire m. c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. Merging, as you proposed, is probably close to the fix for this bug. >> >> And it can't attempt to acquire m while already holding conditionLock, because this would >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while >> already holding m, at line 265. >> >>> >>> >>> WHILE (NOT alerted) AND (NOT waitDone) DO >>> . >>> . >>> . >>> 1 >>> EnterCriticalSection(conditionLock); >>> waitDone := (c.tickets # 0 AND c.counter # count); >> A:---------^ >> B:--------------------------------------------------^ >>> LeaveCriticalSection(conditionLock); >>> END; (* WHILE *) >>> IF waitDone THEN >>> alerted := FALSE; >>> END; >>> m.acquire(); >>> 2 >>> EnterCriticalSection(conditionLock); >>> DEC(c.waiters); >>> IF waitDone THEN esp. here. >>> DEC(c.tickets); >> >> C:---------------^ >>> lastWaiter := (c.tickets = 0); >>> END; >>> LeaveCriticalSection(conditionLock); >>> >>> >>> That is, if we decrement tickets earlier >>> within the first critical section, >>> while we will still wake everyone, >>> only one will decide waitDone and the rest will keep looping. >>> A downside of this, perhaps, is that waking all for Broadcast >>> might be a little slower. >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads >>> implementations, they seem to deal with this differently than us, >>> and they each do about the same thing -- they use a counted semaphore. >>> In the boost case, it appears they duplicate the semaphore for every >>> notification generation, which seems expensive. >>> >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. >>> Or they can do their own reference counting? >>> jdk7 seems to looke like jdk6. >>> The code is gone in jdk8 and I can't easily find the delete in history. >>> The lock merging jdk does probably helps here too. >>> In fact they merge #1 and #2 above as a result. >>> But they still initially wake all threads for signal, not just broadcast. >>> There is also the problem that all the event waits >>> are followed by EnterCriticalSection (or jdk facsimile). >>> - Jay >>> >>> >>> >>> >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> --- >>> From: jay.krell at cornell.edu >>> To: m3devel at elegosoft.com >>> Subject: RE: [M3devel] purported condition variable problems on Win32? >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 >>> >>> Here is another implementation to consider: >>> >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp >>> >>> - Jay >>> >>> >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> --- >>> From: jay.krell at cornell.edu >>> To: m3devel at elegosoft.com >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 >>> Subject: [M3devel] purported condition variable problems on Win32? >>> >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV >>> >>> vs. >>> >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >>> vs. >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain >>> I wrote this: >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; >>> alertable: BOOLEAN) RAISES {Alerted} = >>> (* LL = m on entry and exit, but not for the duration >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp >>> * NOTE that they merge the user lock and the condition lock. >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >>> * "3.3. The Generation Count Solution" >>> *) >>> >>> Do we have the problems described in README.CV? >>> >>> I haven't looked through the ACE code to see >>> to what extent they resemble solution 3.3, or if they >>> changed as a result of this discussion -- which I admit I don't understand >>> and haven't read closely. >>> >>> >>> Spurious wakeups are ok, though should be minimized. >>> >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. >>> >>> Thank you, >>> - Jay >>> >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>> >>> >>> _______________________________________________ >>> M3devel mailing list >>> M3devel at elegosoft.com >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> >> -- >> Rodney Bates >> rodney.m.bates at acm.org >> >> > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Wed Jul 20 20:00:28 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Wed, 20 Jul 2016 18:00:28 -0000 Subject: [M3devel] Yet another bug in Win-32 generation count threads Message-ID: <578FBC29.3000308@lcwb.coop> The more I look at this generation count approach to implementing Modula-3 condition variables on top of what Windows has, the less I like it. It's very hard to understand, but as I get closer, I think I see yet another bug. Assume ticket counting is fixed. There are several waiting threads, say 5. c.waiters = 5 and c.counter = 0. A M3-Broadcast happens. It does (Win-)SetEvent(c.waitEvent), sets c.counter := 1, and c.tickets := 5, making all the waiters eligible to get past the waitDone test, which they start doing. From the abstract M3-condition variable point of view, all have already been released from the Condition variable, but have some cleanup to do inside (M3-)Wait. One by one, they proceed, say 3 of them. Two waiters remain, c.waiters = c.tickets = 2, and all have counts that make them eligible for waitDone. Now, two more waiters do M3-Waits. c.counter still = 1, and their local counts = 1, making them ineligible to get past waitDone. Then a M3-Signal happens. c.counter gets incremented to 2, so all all 4 waiters are now eligible to pass that test. c.tickets gets increased to 3, supposedly so the two that were broadcast-released and one of the new ones can proceed. But it is randomly possible that both the new ones get released (which would be OK), taking two of the tickets, one of the older ones get released, taking the last ticket, and the remaining is stuck waiting when it should not, for lack of a ticket. -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Thu Jul 21 04:00:50 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Thu, 21 Jul 2016 02:00:50 -0000 Subject: [M3devel] Yet another bug in Win-32 generation count threads In-Reply-To: <578FBC29.3000308@lcwb.coop> References: <578FBC29.3000308@lcwb.coop> Message-ID: <57902C9E.1000005@lcwb.coop> If I were to try to fix the Thread condition variables on Windows, I would consider a scheme entirely different from any of those given in Schmidt. Associate the Win-event that a M3-Wait operation uses with a thread, not a condition variable. Since a thread can wait on at most one condition at a time, the Win-event will have at most one waiting thread. This also means allocation of the Win-events is easy: put one in each Activation (which is one-to-one with a thread). (Perhaps the unused waitEvent in an activation was left over from some variant of this Scheme?) Put the Activations of the multiple threads waiting on a Condition in some kind of linked list rooted at the Condition, with link pointers in the Activations. This makes it easy to code almost any wakeup fairness policy in a single place, with single-threaded code: removal from this list, rather than complex interactions among a signaller and many waiter threads. Or, alternatively, it could be done when inserting into the list. Simple FIFO is the obvious choice, but anything could be easily implemented., such as thread priorities. Making the order of reacquisition of the mutex match the order of SetEvent would require more complex work, but shouldn't be too hard. It is an easy-to-maintain invariant that a waiter is waiting on a Win-event iff its Activation is on the list of some condition. This means a signaller will always know a Win-event it is about to call SetEvent on has a thread waiting in it, so can do a self- resetting SetEvent and never worry that the event will remain signalled afterwards. With these restrictions on their use, Win-events behave just like condition variables, greatly simplifying things. All the solutions Schmidt describes suffer from problems originating in having muliple waiters Win-waiting on a single Win-event. One detail that could require some thought is a waiting thread's Activation needs to be removed from its list when the thread is destroyed. On 07/20/2016 01:00 PM, Rodney M. Bates wrote: > The more I look at this generation count approach to implementing > Modula-3 condition variables on top of what Windows has, the less > I like it. It's very hard to understand, but as I get closer, > I think I see yet another bug. > > Assume ticket counting is fixed. > > There are several waiting threads, say 5. c.waiters = 5 and > c.counter = 0. A M3-Broadcast happens. It does > (Win-)SetEvent(c.waitEvent), sets c.counter := 1, and > c.tickets := 5, making all the waiters eligible to get > past the waitDone test, which they start doing. From the > abstract M3-condition variable point of view, all have > already been released from the Condition variable, but > have some cleanup to do inside (M3-)Wait. > > One by one, they proceed, say 3 of them. Two waiters > remain, c.waiters = c.tickets = 2, and all have counts > that make them eligible for waitDone. > > Now, two more waiters do M3-Waits. c.counter still = 1, and > their local counts = 1, making them ineligible to get > past waitDone. Then a M3-Signal happens. c.counter > gets incremented to 2, so all all 4 waiters are now eligible > to pass that test. c.tickets gets increased to 3, supposedly > so the two that were broadcast-released and one of the new > ones can proceed. > > But it is randomly possible that both the new ones get > released (which would be OK), taking two of the tickets, > one of the older ones get released, taking the last ticket, > and the remaining is stuck waiting when it should not, for > lack of a ticket. > -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Thu Jul 21 08:35:45 2016 From: jay.krell at cornell.edu (Jay K) Date: Thu, 21 Jul 2016 06:35:45 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: <578FAA2F.6020105@lcwb.coop> References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop> Message-ID: Am I understanding this? There are some waiters. There is a broadcast ("wake all"). Some waiters are woken. There are more waiters. There is a signal ("wake one"). There is a chance that some of the original waiters will remain waiting while the later waiters are not. That is, the signaled threads kind of steal the wake intended by the broadcast. Is this unfairness or incorrectness? It feels like unfairness. It feels like condition variables are specified fairly weakly. - Jay > Date: Wed, 20 Jul 2016 11:43:27 -0500 > From: rodney_bates at lcwb.coop > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] purported condition variable problems on Win32? > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > This gets confusing, since there are Modula-3 waits, signals, > etc. and similar concepts in Win32, but which are not the > same. I am going the try to put "M3-" and "Win-" prefixes on > things to make this clearer. I cartainly need to do this > for my own benefit, if for nobody else. > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > c.tickets = 0, and c.waiters = 2. > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > which will allow both the waiters to Win-wakeup sometime soon. > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > to eventually allow only one waiter to M3-wakeup) and increments > c.counter, setting things up so that the waitDone expression at :308 > will be TRUE for the waiting threads, the next time they evaluate it. > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > releases c.lock, and tries to acquire m. The problem arises > because the second waiter can get to these steps before the first > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > second to reach :308 will find c.tickets # 0, as did the first, > and proceed. > > Each will eventually, do a M3-wake. If this were Posix condition > variables, this would be wrong, because Posix says only one > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > says one or more waiters proceed. However, each will, in the > process, get to :322, and DEC(c.tickets). Signal only provided > one ticket, but the waiters have taken two. c.tickets goes > negative. > > After this, when some thread can do another (M3-)Wait, and some > other does another (M3-)Signal, c.tickets increments only back to > zero, which means the new waiter will not M3-wake, even though it > should. > > > I am going to try to construct a test case that will force > this scenario. > > See more below: > > > On 07/18/2016 08:53 PM, Jay wrote: > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > Otherwise previous version I think had a very large lock. > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > I guess you are saying it would be unfair, which is ok. > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > I'll check on the unused event. > > > > - Jay > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > >> > >> > >> > >> > >> > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > >> comments: > >> > >> This looks like a very direct implementation of Schmidts' generation count > >> algorithm, with a number of important little details additionally taken care > >> of. > >> > >> On small difference is at line 308 (B, below) where Schmidt does > >> c.counter>count instead of #. Usually, this would make no difference, > >> since counter only increases, but on a long-running program, it could > >> overflow. If this happened,as it often does, as silent wrap-around to > >> FIRST(INTEGER), some threads could be trapped for an extremely long > >> time, waiting for counter come back around greater than their count. As > >> coded in ThreadWin32, they would be allowed to proceed normally. > >> > >> The only glitch would be that, if there were simultaneously waiting threads > >> separated by NUMBER(INTEGER) truly different generations (but the same value > >> of c.counter), the recent ones could unfairly proceed along with the > >> extremely old ones. That seems extremely less likely than the > >> mere existence of an overflow. > >> > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > >> only created at :537 and deleted at :554, but never used in any real > >> way. It could be deleted. > >> > >> As for this code: > >> > >> EnterCriticalSection(conditionLock); > >> > >> (* Capture the value of the counter before we start waiting. > >> * We will not stop waiting until the counter changes. > >> * That is, we will not stop waiting until a signal > >> * comes in after we start waiting. > >> *) > >> > >> count := c.counter; > >> INC(c.waiters); > >> > >> LeaveCriticalSection(conditionLock); > >> m.release(); (* can this be moved to before taking conditionLock? *) > >> > >> No, it is important to sample and save c.counter in the order threads wait. > >> Retaining m until this is done ensures this. Otherwise, some other thread > >> that waited later could have altered c.counter before this one gets conditionLock > >> and saves its copy of c.counter. > >> > >> I have made several comment changes to ThreadWin32.m3 that would have made > >> it easier to vet. I will commit these, but only comments. It is either > >> very difficult or impossible for me to test or even recompile this module, > >> and, especially as fragile as this kind of code is, I don't want to commit > >> any substantive changes untested and uncompiled. > >> > >> More below: > >> > >>> On 07/07/2016 04:24 AM, Jay K wrote: > >>> So...I do NOT understand all of this. > >>> > >>> > >>> However, comparing the three implementations > >>> and attempting to understand ours, > >>> I am struck by the following > >>> > >>> - Our broadcast seems ok, but > >>> - our signal seems to wake everyone, > >>> and then...they race a bit, > >>> one will decide it is last, and > >>> reset the event, but every prior waiter > >>> is still woken. > >>> Why not merge the following chunks: > >> > >> I don't think this will work. It is important that, once a thread has decided, > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > >> if it is also the last of its generation to proceed, resets c.waitevent. This > >> is ensured by its having already reacquired m first, preventing any other thread > >> from returning from its Wait yet. > > I was wrong about this. There is already no fairness being enforced here. Even with > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > Merging, as you proposed, is probably close to the fix for this bug. > > > >> > >> And it can't attempt to acquire m while already holding conditionLock, because this would > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > >> already holding m, at line 265. > >> > >>> > >>> > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > >>> . > >>> . > >>> . > >>> 1 > >>> EnterCriticalSection(conditionLock); > >>> waitDone := (c.tickets # 0 AND c.counter # count); > >> A:---------^ > >> B:--------------------------------------------------^ > >>> LeaveCriticalSection(conditionLock); > >>> END; (* WHILE *) > >>> IF waitDone THEN > >>> alerted := FALSE; > >>> END; > >>> m.acquire(); > >>> 2 > >>> EnterCriticalSection(conditionLock); > >>> DEC(c.waiters); > >>> IF waitDone THEN esp. here. > >>> DEC(c.tickets); > >> > >> C:---------------^ > >>> lastWaiter := (c.tickets = 0); > >>> END; > >>> LeaveCriticalSection(conditionLock); > >>> > >>> > >>> That is, if we decrement tickets earlier > >>> within the first critical section, > >>> while we will still wake everyone, > >>> only one will decide waitDone and the rest will keep looping. > >>> A downside of this, perhaps, is that waking all for Broadcast > >>> might be a little slower. > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > >>> implementations, they seem to deal with this differently than us, > >>> and they each do about the same thing -- they use a counted semaphore. > >>> In the boost case, it appears they duplicate the semaphore for every > >>> notification generation, which seems expensive. > >>> > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > >>> Or they can do their own reference counting? > >>> jdk7 seems to looke like jdk6. > >>> The code is gone in jdk8 and I can't easily find the delete in history. > >>> The lock merging jdk does probably helps here too. > >>> In fact they merge #1 and #2 above as a result. > >>> But they still initially wake all threads for signal, not just broadcast. > >>> There is also the problem that all the event waits > >>> are followed by EnterCriticalSection (or jdk facsimile). > >>> - Jay > >>> > >>> > >>> > >>> > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! > >> --- > >>> From: jay.krell at cornell.edu > >>> To: m3devel at elegosoft.com > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > >>> > >>> Here is another implementation to consider: > >>> > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > >>> > >>> - Jay > >>> > >>> > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! > >> --- > >>> From: jay.krell at cornell.edu > >>> To: m3devel at elegosoft.com > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > >>> Subject: [M3devel] purported condition variable problems on Win32? > >>> > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > >>> > >>> vs. > >>> > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >>> vs. > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > >>> I wrote this: > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > >>> alertable: BOOLEAN) RAISES {Alerted} = > >>> (* LL = m on entry and exit, but not for the duration > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > >>> * NOTE that they merge the user lock and the condition lock. > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >>> * "3.3. The Generation Count Solution" > >>> *) > >>> > >>> Do we have the problems described in README.CV? > >>> > >>> I haven't looked through the ACE code to see > >>> to what extent they resemble solution 3.3, or if they > >>> changed as a result of this discussion -- which I admit I don't understand > >>> and haven't read closely. > >>> > >>> > >>> Spurious wakeups are ok, though should be minimized. > >>> > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > >>> > >>> Thank you, > >>> - Jay > >>> > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >>> > >>> > >>> _______________________________________________ > >>> M3devel mailing list > >>> M3devel at elegosoft.com > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >> > >> -- > >> Rodney Bates > >> rodney.m.bates at acm.org > >> > >> > > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Wed Jul 20 17:36:13 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Wed, 20 Jul 2016 15:36:13 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop> <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> Message-ID: <578F99C5.3010702@lcwb.coop> On 07/18/2016 08:53 PM, Jay wrote: > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > Otherwise previous version I think had a very large lock. > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > I guess you are saying it would be unfair, which is ok. I think ths is OK as coded. The # will, overwhelmingly often, make wraparound work as expected, whereas the > would leave threads trapped at the upper end for ~ FIRST(INTEGER) generations. I am not sure about the cases that have problems with #, but for anything to happen at all, there would have to be a wrap, further followed by a thread that got starved for another ~ NUMBER(INTEGER) generations of other threads getting through. Even at 32 bits, this is extremely unlikely, and probably something else would fail before this. If it really were a problem, I think counter can be reset to zero any time there are no waiters, which would usually be very often. > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > I'll check on the unused event. > > - Jay > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: >> >> >> >> >> >> I spent some time looking over the signal part of ThreadWin32.m3. A few >> comments: >> >> This looks like a very direct implementation of Schmidts' generation count >> algorithm, with a number of important little details additionally taken care >> of. >> >> On small difference is at line 308 (B, below) where Schmidt does >> c.counter>count instead of #. Usually, this would make no difference, >> since counter only increases, but on a long-running program, it could >> overflow. If this happened,as it often does, as silent wrap-around to >> FIRST(INTEGER), some threads could be trapped for an extremely long >> time, waiting for counter come back around greater than their count. As >> coded in ThreadWin32, they would be allowed to proceed normally. >> >> The only glitch would be that, if there were simultaneously waiting threads >> separated by NUMBER(INTEGER) truly different generations (but the same value >> of c.counter), the recent ones could unfairly proceed along with the >> extremely old ones. That seems extremely less likely than the >> mere existence of an overflow. >> >> Both type Condition and type Activation have a field waitEvent: HANDLE. >> Condition.waitEvent is extensively used, but Activation.waitEvent is >> only created at :537 and deleted at :554, but never used in any real >> way. It could be deleted. >> >> As for this code: >> >> EnterCriticalSection(conditionLock); >> >> (* Capture the value of the counter before we start waiting. >> * We will not stop waiting until the counter changes. >> * That is, we will not stop waiting until a signal >> * comes in after we start waiting. >> *) >> >> count := c.counter; >> INC(c.waiters); >> >> LeaveCriticalSection(conditionLock); >> m.release(); (* can this be moved to before taking conditionLock? *) >> >> No, it is important to sample and save c.counter in the order threads wait. >> Retaining m until this is done ensures this. Otherwise, some other thread >> that waited later could have altered c.counter before this one gets conditionLock >> and saves its copy of c.counter. >> >> I have made several comment changes to ThreadWin32.m3 that would have made >> it easier to vet. I will commit these, but only comments. It is either >> very difficult or impossible for me to test or even recompile this module, >> and, especially as fragile as this kind of code is, I don't want to commit >> any substantive changes untested and uncompiled. >> >> More below: >> >>> On 07/07/2016 04:24 AM, Jay K wrote: >>> So...I do NOT understand all of this. >>> >>> >>> However, comparing the three implementations >>> and attempting to understand ours, >>> I am struck by the following >>> >>> - Our broadcast seems ok, but >>> - our signal seems to wake everyone, >>> and then...they race a bit, >>> one will decide it is last, and >>> reset the event, but every prior waiter >>> is still woken. >>> Why not merge the following chunks: >> >> I don't think this will work. It is important that, once a thread has decided, >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, >> if it is also the last of its generation to proceed, resets c.waitevent. This >> is ensured by its having already reacquired m first, preventing any other thread >> from returning from its Wait yet. >> >> And it can't attempt to acquire m while already holding conditionLock, because this would >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while >> already holding m, at line 265. >> >>> >>> >>> WHILE (NOT alerted) AND (NOT waitDone) DO >>> . >>> . >>> . >>> 1 >>> EnterCriticalSection(conditionLock); >>> waitDone := (c.tickets # 0 AND c.counter # count); >> A:---------^ >> B:--------------------------------------------------^ >>> LeaveCriticalSection(conditionLock); >>> END; (* WHILE *) >>> IF waitDone THEN >>> alerted := FALSE; >>> END; >>> m.acquire(); >>> 2 >>> EnterCriticalSection(conditionLock); >>> DEC(c.waiters); >>> IF waitDone THEN esp. here. >>> DEC(c.tickets); >> >> C:---------------^ >>> lastWaiter := (c.tickets = 0); >>> END; >>> LeaveCriticalSection(conditionLock); >>> >>> >>> That is, if we decrement tickets earlier >>> within the first critical section, >>> while we will still wake everyone, >>> only one will decide waitDone and the rest will keep looping. >>> A downside of this, perhaps, is that waking all for Broadcast >>> might be a little slower. >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads >>> implementations, they seem to deal with this differently than us, >>> and they each do about the same thing -- they use a counted semaphore. >>> In the boost case, it appears they duplicate the semaphore for every >>> notification generation, which seems expensive. >>> >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. >>> Or they can do their own reference counting? >>> jdk7 seems to looke like jdk6. >>> The code is gone in jdk8 and I can't easily find the delete in history. >>> The lock merging jdk does probably helps here too. >>> In fact they merge #1 and #2 above as a result. >>> But they still initially wake all threads for signal, not just broadcast. >>> There is also the problem that all the event waits >>> are followed by EnterCriticalSection (or jdk facsimile). >>> - Jay >>> >>> >>> >>> >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> --- >>> From: jay.krell at cornell.edu >>> To: m3devel at elegosoft.com >>> Subject: RE: [M3devel] purported condition variable problems on Win32? >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 >>> >>> Here is another implementation to consider: >>> >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp >>> >>> - Jay >>> >>> >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> --- >>> From: jay.krell at cornell.edu >>> To: m3devel at elegosoft.com >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 >>> Subject: [M3devel] purported condition variable problems on Win32? >>> >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV >>> >>> vs. >>> >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >>> vs. >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain >>> I wrote this: >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; >>> alertable: BOOLEAN) RAISES {Alerted} = >>> (* LL = m on entry and exit, but not for the duration >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp >>> * NOTE that they merge the user lock and the condition lock. >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >>> * "3.3. The Generation Count Solution" >>> *) >>> >>> Do we have the problems described in README.CV? >>> >>> I haven't looked through the ACE code to see >>> to what extent they resemble solution 3.3, or if they >>> changed as a result of this discussion -- which I admit I don't understand >>> and haven't read closely. >>> >>> >>> Spurious wakeups are ok, though should be minimized. >>> >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. >>> >>> Thank you, >>> - Jay >>> >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>> >>> >>> _______________________________________________ >>> M3devel mailing list >>> M3devel at elegosoft.com >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> >> -- >> Rodney Bates >> rodney.m.bates at acm.org >> >> > -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Thu Jul 21 08:43:20 2016 From: jay.krell at cornell.edu (Jay K) Date: Thu, 21 Jul 2016 06:43:20 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop>, <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop>, Message-ID: That question was meant for the other bug. Can this one be fixed by rechecking if tickets is positive before decrementing it? This seems like a big problem in Schmidt's code. I'm nervous about merging the locks, but maybe. In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. Though I realize that is actually necessarily part of what gets done. Schmidt clearly did not merge the locks, and the Java version does. The Java version also has its own simple and not-terrible critical section. The Java version also disappeared in newer releases and I didn't track down what happened to it. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org CC: m3devel at elegosoft.com Subject: RE: [M3devel] purported condition variable problems on Win32? Date: Thu, 21 Jul 2016 06:35:41 +0000 Am I understanding this? There are some waiters. There is a broadcast ("wake all"). Some waiters are woken. There are more waiters. There is a signal ("wake one"). There is a chance that some of the original waiters will remain waiting while the later waiters are not. That is, the signaled threads kind of steal the wake intended by the broadcast. Is this unfairness or incorrectness? It feels like unfairness. It feels like condition variables are specified fairly weakly. - Jay > Date: Wed, 20 Jul 2016 11:43:27 -0500 > From: rodney_bates at lcwb.coop > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] purported condition variable problems on Win32? > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > This gets confusing, since there are Modula-3 waits, signals, > etc. and similar concepts in Win32, but which are not the > same. I am going the try to put "M3-" and "Win-" prefixes on > things to make this clearer. I cartainly need to do this > for my own benefit, if for nobody else. > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > c.tickets = 0, and c.waiters = 2. > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > which will allow both the waiters to Win-wakeup sometime soon. > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > to eventually allow only one waiter to M3-wakeup) and increments > c.counter, setting things up so that the waitDone expression at :308 > will be TRUE for the waiting threads, the next time they evaluate it. > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > releases c.lock, and tries to acquire m. The problem arises > because the second waiter can get to these steps before the first > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > second to reach :308 will find c.tickets # 0, as did the first, > and proceed. > > Each will eventually, do a M3-wake. If this were Posix condition > variables, this would be wrong, because Posix says only one > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > says one or more waiters proceed. However, each will, in the > process, get to :322, and DEC(c.tickets). Signal only provided > one ticket, but the waiters have taken two. c.tickets goes > negative. > > After this, when some thread can do another (M3-)Wait, and some > other does another (M3-)Signal, c.tickets increments only back to > zero, which means the new waiter will not M3-wake, even though it > should. > > > I am going to try to construct a test case that will force > this scenario. > > See more below: > > > On 07/18/2016 08:53 PM, Jay wrote: > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > Otherwise previous version I think had a very large lock. > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > I guess you are saying it would be unfair, which is ok. > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > I'll check on the unused event. > > > > - Jay > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > >> > >> > >> > >> > >> > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > >> comments: > >> > >> This looks like a very direct implementation of Schmidts' generation count > >> algorithm, with a number of important little details additionally taken care > >> of. > >> > >> On small difference is at line 308 (B, below) where Schmidt does > >> c.counter>count instead of #. Usually, this would make no difference, > >> since counter only increases, but on a long-running program, it could > >> overflow. If this happened,as it often does, as silent wrap-around to > >> FIRST(INTEGER), some threads could be trapped for an extremely long > >> time, waiting for counter come back around greater than their count. As > >> coded in ThreadWin32, they would be allowed to proceed normally. > >> > >> The only glitch would be that, if there were simultaneously waiting threads > >> separated by NUMBER(INTEGER) truly different generations (but the same value > >> of c.counter), the recent ones could unfairly proceed along with the > >> extremely old ones. That seems extremely less likely than the > >> mere existence of an overflow. > >> > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > >> only created at :537 and deleted at :554, but never used in any real > >> way. It could be deleted. > >> > >> As for this code: > >> > >> EnterCriticalSection(conditionLock); > >> > >> (* Capture the value of the counter before we start waiting. > >> * We will not stop waiting until the counter changes. > >> * That is, we will not stop waiting until a signal > >> * comes in after we start waiting. > >> *) > >> > >> count := c.counter; > >> INC(c.waiters); > >> > >> LeaveCriticalSection(conditionLock); > >> m.release(); (* can this be moved to before taking conditionLock? *) > >> > >> No, it is important to sample and save c.counter in the order threads wait. > >> Retaining m until this is done ensures this. Otherwise, some other thread > >> that waited later could have altered c.counter before this one gets conditionLock > >> and saves its copy of c.counter. > >> > >> I have made several comment changes to ThreadWin32.m3 that would have made > >> it easier to vet. I will commit these, but only comments. It is either > >> very difficult or impossible for me to test or even recompile this module, > >> and, especially as fragile as this kind of code is, I don't want to commit > >> any substantive changes untested and uncompiled. > >> > >> More below: > >> > >>> On 07/07/2016 04:24 AM, Jay K wrote: > >>> So...I do NOT understand all of this. > >>> > >>> > >>> However, comparing the three implementations > >>> and attempting to understand ours, > >>> I am struck by the following > >>> > >>> - Our broadcast seems ok, but > >>> - our signal seems to wake everyone, > >>> and then...they race a bit, > >>> one will decide it is last, and > >>> reset the event, but every prior waiter > >>> is still woken. > >>> Why not merge the following chunks: > >> > >> I don't think this will work. It is important that, once a thread has decided, > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > >> if it is also the last of its generation to proceed, resets c.waitevent. This > >> is ensured by its having already reacquired m first, preventing any other thread > >> from returning from its Wait yet. > > I was wrong about this. There is already no fairness being enforced here. Even with > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > Merging, as you proposed, is probably close to the fix for this bug. > > > >> > >> And it can't attempt to acquire m while already holding conditionLock, because this would > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > >> already holding m, at line 265. > >> > >>> > >>> > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > >>> . > >>> . > >>> . > >>> 1 > >>> EnterCriticalSection(conditionLock); > >>> waitDone := (c.tickets # 0 AND c.counter # count); > >> A:---------^ > >> B:--------------------------------------------------^ > >>> LeaveCriticalSection(conditionLock); > >>> END; (* WHILE *) > >>> IF waitDone THEN > >>> alerted := FALSE; > >>> END; > >>> m.acquire(); > >>> 2 > >>> EnterCriticalSection(conditionLock); > >>> DEC(c.waiters); > >>> IF waitDone THEN esp. here. > >>> DEC(c.tickets); > >> > >> C:---------------^ > >>> lastWaiter := (c.tickets = 0); > >>> END; > >>> LeaveCriticalSection(conditionLock); > >>> > >>> > >>> That is, if we decrement tickets earlier > >>> within the first critical section, > >>> while we will still wake everyone, > >>> only one will decide waitDone and the rest will keep looping. > >>> A downside of this, perhaps, is that waking all for Broadcast > >>> might be a little slower. > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > >>> implementations, they seem to deal with this differently than us, > >>> and they each do about the same thing -- they use a counted semaphore. > >>> In the boost case, it appears they duplicate the semaphore for every > >>> notification generation, which seems expensive. > >>> > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > >>> Or they can do their own reference counting? > >>> jdk7 seems to looke like jdk6. > >>> The code is gone in jdk8 and I can't easily find the delete in history. > >>> The lock merging jdk does probably helps here too. > >>> In fact they merge #1 and #2 above as a result. > >>> But they still initially wake all threads for signal, not just broadcast. > >>> There is also the problem that all the event waits > >>> are followed by EnterCriticalSection (or jdk facsimile). > >>> - Jay > >>> > >>> > >>> > >>> > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! > >> --- > >>> From: jay.krell at cornell.edu > >>> To: m3devel at elegosoft.com > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > >>> > >>> Here is another implementation to consider: > >>> > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > >>> > >>> - Jay > >>> > >>> > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! > >> --- > >>> From: jay.krell at cornell.edu > >>> To: m3devel at elegosoft.com > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > >>> Subject: [M3devel] purported condition variable problems on Win32? > >>> > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > >>> > >>> vs. > >>> > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >>> vs. > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > >>> I wrote this: > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > >>> alertable: BOOLEAN) RAISES {Alerted} = > >>> (* LL = m on entry and exit, but not for the duration > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > >>> * NOTE that they merge the user lock and the condition lock. > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >>> * "3.3. The Generation Count Solution" > >>> *) > >>> > >>> Do we have the problems described in README.CV? > >>> > >>> I haven't looked through the ACE code to see > >>> to what extent they resemble solution 3.3, or if they > >>> changed as a result of this discussion -- which I admit I don't understand > >>> and haven't read closely. > >>> > >>> > >>> Spurious wakeups are ok, though should be minimized. > >>> > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > >>> > >>> Thank you, > >>> - Jay > >>> > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >>> > >>> > >>> _______________________________________________ > >>> M3devel mailing list > >>> M3devel at elegosoft.com > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >> > >> -- > >> Rodney Bates > >> rodney.m.bates at acm.org > >> > >> > > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Thu Jul 21 19:01:38 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Thu, 21 Jul 2016 17:01:38 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop> Message-ID: <5790FF55.5030007@lcwb.coop> On 07/21/2016 01:35 AM, Jay K wrote: > Am I understanding this? > There are some waiters. > There is a broadcast ("wake all"). Some waiters are woken. > There are more waiters. > There is a signal ("wake one"). > There is a chance that some of the original waiters will remain waiting while the later waiters are not. > That is, the signaled threads kind of steal the wake intended by the broadcast. > > > Is this unfairness or incorrectness? > It feels like unfairness. > It feels like condition variables are specified fairly weakly. > It's incorrectness. The older waiters were waiting at the time of the Broadcast and should all M3-awaken, but one does not. As a aside, more than one of the newer waiters M3-awakens in response to the Signal, which is more than necessary, but OK, the way M3 Signal is specified. It's all because the ticket mechanism only controls the number of treads to wake up, but, in this case, it matters not just how many but which one(s). > > - Jay > > > > Date: Wed, 20 Jul 2016 11:43:27 -0500 > > From: rodney_bates at lcwb.coop > > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] purported condition variable problems on Win32? > > > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > > > This gets confusing, since there are Modula-3 waits, signals, > > etc. and similar concepts in Win32, but which are not the > > same. I am going the try to put "M3-" and "Win-" prefixes on > > things to make this clearer. I cartainly need to do this > > for my own benefit, if for nobody else. > > > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > > c.tickets = 0, and c.waiters = 2. > > > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > > which will allow both the waiters to Win-wakeup sometime soon. > > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > > to eventually allow only one waiter to M3-wakeup) and increments > > c.counter, setting things up so that the waitDone expression at :308 > > will be TRUE for the waiting threads, the next time they evaluate it. > > > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > > releases c.lock, and tries to acquire m. The problem arises > > because the second waiter can get to these steps before the first > > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > > second to reach :308 will find c.tickets # 0, as did the first, > > and proceed. > > > > Each will eventually, do a M3-wake. If this were Posix condition > > variables, this would be wrong, because Posix says only one > > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > > says one or more waiters proceed. However, each will, in the > > process, get to :322, and DEC(c.tickets). Signal only provided > > one ticket, but the waiters have taken two. c.tickets goes > > negative. > > > > After this, when some thread can do another (M3-)Wait, and some > > other does another (M3-)Signal, c.tickets increments only back to > > zero, which means the new waiter will not M3-wake, even though it > > should. > > > > > > I am going to try to construct a test case that will force > > this scenario. > > > > See more below: > > > > > > On 07/18/2016 08:53 PM, Jay wrote: > > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > > > Otherwise previous version I think had a very large lock. > > > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > > > I guess you are saying it would be unfair, which is ok. > > > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > > > I'll check on the unused event. > > > > > > - Jay > > > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > > >> > > >> > > >> > > >> > > >> > > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > > >> comments: > > >> > > >> This looks like a very direct implementation of Schmidts' generation count > > >> algorithm, with a number of important little details additionally taken care > > >> of. > > >> > > >> On small difference is at line 308 (B, below) where Schmidt does > > >> c.counter>count instead of #. Usually, this would make no difference, > > >> since counter only increases, but on a long-running program, it could > > >> overflow. If this happened,as it often does, as silent wrap-around to > > >> FIRST(INTEGER), some threads could be trapped for an extremely long > > >> time, waiting for counter come back around greater than their count. As > > >> coded in ThreadWin32, they would be allowed to proceed normally. > > >> > > >> The only glitch would be that, if there were simultaneously waiting threads > > >> separated by NUMBER(INTEGER) truly different generations (but the same value > > >> of c.counter), the recent ones could unfairly proceed along with the > > >> extremely old ones. That seems extremely less likely than the > > >> mere existence of an overflow. > > >> > > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > > >> only created at :537 and deleted at :554, but never used in any real > > >> way. It could be deleted. > > >> > > >> As for this code: > > >> > > >> EnterCriticalSection(conditionLock); > > >> > > >> (* Capture the value of the counter before we start waiting. > > >> * We will not stop waiting until the counter changes. > > >> * That is, we will not stop waiting until a signal > > >> * comes in after we start waiting. > > >> *) > > >> > > >> count := c.counter; > > >> INC(c.waiters); > > >> > > >> LeaveCriticalSection(conditionLock); > > >> m.release(); (* can this be moved to before taking conditionLock? *) > > >> > > >> No, it is important to sample and save c.counter in the order threads wait. > > >> Retaining m until this is done ensures this. Otherwise, some other thread > > >> that waited later could have altered c.counter before this one gets conditionLock > > >> and saves its copy of c.counter. > > >> > > >> I have made several comment changes to ThreadWin32.m3 that would have made > > >> it easier to vet. I will commit these, but only comments. It is either > > >> very difficult or impossible for me to test or even recompile this module, > > >> and, especially as fragile as this kind of code is, I don't want to commit > > >> any substantive changes untested and uncompiled. > > >> > > >> More below: > > >> > > >>> On 07/07/2016 04:24 AM, Jay K wrote: > > >>> So...I do NOT understand all of this. > > >>> > > >>> > > >>> However, comparing the three implementations > > >>> and attempting to understand ours, > > >>> I am struck by the following > > >>> > > >>> - Our broadcast seems ok, but > > >>> - our signal seems to wake everyone, > > >>> and then...they race a bit, > > >>> one will decide it is last, and > > >>> reset the event, but every prior waiter > > >>> is still woken. > > >>> Why not merge the following chunks: > > >> > > >> I don't think this will work. It is important that, once a thread has decided, > > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > > >> if it is also the last of its generation to proceed, resets c.waitevent. This > > >> is ensured by its having already reacquired m first, preventing any other thread > > >> from returning from its Wait yet. > > > > I was wrong about this. There is already no fairness being enforced here. Even with > > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > > > Merging, as you proposed, is probably close to the fix for this bug. > > > > > > >> > > >> And it can't attempt to acquire m while already holding conditionLock, because this would > > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > > >> already holding m, at line 265. > > >> > > >>> > > >>> > > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > > >>> . > > >>> . > > >>> . > > >>> 1 > > >>> EnterCriticalSection(conditionLock); > > >>> waitDone := (c.tickets # 0 AND c.counter # count); > > >> A:---------^ > > >> B:--------------------------------------------------^ > > >>> LeaveCriticalSection(conditionLock); > > >>> END; (* WHILE *) > > >>> IF waitDone THEN > > >>> alerted := FALSE; > > >>> END; > > >>> m.acquire(); > > >>> 2 > > >>> EnterCriticalSection(conditionLock); > > >>> DEC(c.waiters); > > >>> IF waitDone THEN esp. here. > > >>> DEC(c.tickets); > > >> > > >> C:---------------^ > > >>> lastWaiter := (c.tickets = 0); > > >>> END; > > >>> LeaveCriticalSection(conditionLock); > > >>> > > >>> > > >>> That is, if we decrement tickets earlier > > >>> within the first critical section, > > >>> while we will still wake everyone, > > >>> only one will decide waitDone and the rest will keep looping. > > >>> A downside of this, perhaps, is that waking all for Broadcast > > >>> might be a little slower. > > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > > >>> implementations, they seem to deal with this differently than us, > > >>> and they each do about the same thing -- they use a counted semaphore. > > >>> In the boost case, it appears they duplicate the semaphore for every > > >>> notification generation, which seems expensive. > > >>> > > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > > >>> Or they can do their own reference counting? > > >>> jdk7 seems to looke like jdk6. > > >>> The code is gone in jdk8 and I can't easily find the delete in history. > > >>> The lock merging jdk does probably helps here too. > > >>> In fact they merge #1 and #2 above as a result. > > >>> But they still initially wake all threads for signal, not just broadcast. > > >>> There is also the problem that all the event waits > > >>> are followed by EnterCriticalSection (or jdk facsimile). > > >>> - Jay > > >>> > > >>> > > >>> > > >>> > > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > --! > > >> --- > > >>> From: jay.krell at cornell.edu > > >>> To: m3devel at elegosoft.com > > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > > >>> > > >>> Here is another implementation to consider: > > >>> > > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > > >>> > > >>> - Jay > > >>> > > >>> > > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > --! > > >> --- > > >>> From: jay.krell at cornell.edu > > >>> To: m3devel at elegosoft.com > > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > > >>> Subject: [M3devel] purported condition variable problems on Win32? > > >>> > > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > > >>> > > >>> vs. > > >>> > > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > >>> vs. > > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > > >>> I wrote this: > > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > > >>> alertable: BOOLEAN) RAISES {Alerted} = > > >>> (* LL = m on entry and exit, but not for the duration > > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > > >>> * NOTE that they merge the user lock and the condition lock. > > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > >>> * "3.3. The Generation Count Solution" > > >>> *) > > >>> > > >>> Do we have the problems described in README.CV? > > >>> > > >>> I haven't looked through the ACE code to see > > >>> to what extent they resemble solution 3.3, or if they > > >>> changed as a result of this discussion -- which I admit I don't understand > > >>> and haven't read closely. > > >>> > > >>> > > >>> Spurious wakeups are ok, though should be minimized. > > >>> > > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > >>> > > >>> Thank you, > > >>> - Jay > > >>> > > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > >>> > > >>> > > >>> _______________________________________________ > > >>> M3devel mailing list > > >>> M3devel at elegosoft.com > > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > >> > > >> -- > > >> Rodney Bates > > >> rodney.m.bates at acm.org > > >> > > >> > > > > > > > -- > > Rodney Bates > > rodney.m.bates at acm.org -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Thu Jul 21 19:04:40 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Thu, 21 Jul 2016 17:04:40 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop>, <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop>, Message-ID: <57910079.6090705@lcwb.coop> I think the first lost-wakeup bug can be fixed by merging the two lock sections on c.lock or maybe moving just the DEC(c.tickets) up to the first. I would need to write sample code, then review it to be confident. I have not done that so far. Maybe the Java folks knew this and got it right. On 07/21/2016 01:43 AM, Jay K wrote: > That question was meant for the other bug. > > Can this one be fixed by rechecking if tickets is positive before decrementing it? > This seems like a big problem in Schmidt's code. > > I'm nervous about merging the locks, but maybe. > In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. > Though I realize that is actually necessarily part of what gets done. > Schmidt clearly did not merge the locks, and the Java version does. > The Java version also has its own simple and not-terrible critical section. > The Java version also disappeared in newer releases and I didn't track down what happened to it. > > - Jay > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > From: jay.krell at cornell.edu > To: rodney.m.bates at acm.org > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] purported condition variable problems on Win32? > Date: Thu, 21 Jul 2016 06:35:41 +0000 > > Am I understanding this? > There are some waiters. > There is a broadcast ("wake all"). Some waiters are woken. > There are more waiters. > There is a signal ("wake one"). > There is a chance that some of the original waiters will remain waiting while the later waiters are not. > That is, the signaled threads kind of steal the wake intended by the broadcast. > > > Is this unfairness or incorrectness? > It feels like unfairness. > It feels like condition variables are specified fairly weakly. > > > - Jay > > > > Date: Wed, 20 Jul 2016 11:43:27 -0500 > > From: rodney_bates at lcwb.coop > > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] purported condition variable problems on Win32? > > > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > > > This gets confusing, since there are Modula-3 waits, signals, > > etc. and similar concepts in Win32, but which are not the > > same. I am going the try to put "M3-" and "Win-" prefixes on > > things to make this clearer. I cartainly need to do this > > for my own benefit, if for nobody else. > > > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > > c.tickets = 0, and c.waiters = 2. > > > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > > which will allow both the waiters to Win-wakeup sometime soon. > > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > > to eventually allow only one waiter to M3-wakeup) and increments > > c.counter, setting things up so that the waitDone expression at :308 > > will be TRUE for the waiting threads, the next time they evaluate it. > > > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > > releases c.lock, and tries to acquire m. The problem arises > > because the second waiter can get to these steps before the first > > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > > second to reach :308 will find c.tickets # 0, as did the first, > > and proceed. > > > > Each will eventually, do a M3-wake. If this were Posix condition > > variables, this would be wrong, because Posix says only one > > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > > says one or more waiters proceed. However, each will, in the > > process, get to :322, and DEC(c.tickets). Signal only provided > > one ticket, but the waiters have taken two. c.tickets goes > > negative. > > > > After this, when some thread can do another (M3-)Wait, and some > > other does another (M3-)Signal, c.tickets increments only back to > > zero, which means the new waiter will not M3-wake, even though it > > should. > > > > > > I am going to try to construct a test case that will force > > this scenario. > > > > See more below: > > > > > > On 07/18/2016 08:53 PM, Jay wrote: > > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > > > Otherwise previous version I think had a very large lock. > > > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > > > I guess you are saying it would be unfair, which is ok. > > > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > > > I'll check on the unused event. > > > > > > - Jay > > > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > > >> > > >> > > >> > > >> > > >> > > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > > >> comments: > > >> > > >> This looks like a very direct implementation of Schmidts' generation count > > >> algorithm, with a number of important little details additionally taken care > > >> of. > > >> > > >> On small difference is at line 308 (B, below) where Schmidt does > > >> c.counter>count instead of #. Usually, this would make no difference, > > >> since counter only increases, but on a long-running program, it could > > >> overflow. If this happened,as it often does, as silent wrap-around to > > >> FIRST(INTEGER), some threads could be trapped for an extremely long > > >> time, waiting for counter come back around greater than their count. As > > >> coded in ThreadWin32, they would be allowed to proceed normally. > > >> > > >> The only glitch would be that, if there were simultaneously waiting threads > > >> separated by NUMBER(INTEGER) truly different generations (but the same value > > >> of c.counter), the recent ones could unfairly proceed along with the > > >> extremely old ones. That seems extremely less likely than the > > >> mere existence of an overflow. > > >> > > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > > >> only created at :537 and deleted at :554, but never used in any real > > >> way. It could be deleted. > > >> > > >> As for this code: > > >> > > >> EnterCriticalSection(conditionLock); > > >> > > >> (* Capture the value of the counter before we start waiting. > > >> * We will not stop waiting until the counter changes. > > >> * That is, we will not stop waiting until a signal > > >> * comes in after we start waiting. > > >> *) > > >> > > >> count := c.counter; > > >> INC(c.waiters); > > >> > > >> LeaveCriticalSection(conditionLock); > > >> m.release(); (* can this be moved to before taking conditionLock? *) > > >> > > >> No, it is important to sample and save c.counter in the order threads wait. > > >> Retaining m until this is done ensures this. Otherwise, some other thread > > >> that waited later could have altered c.counter before this one gets conditionLock > > >> and saves its copy of c.counter. > > >> > > >> I have made several comment changes to ThreadWin32.m3 that would have made > > >> it easier to vet. I will commit these, but only comments. It is either > > >> very difficult or impossible for me to test or even recompile this module, > > >> and, especially as fragile as this kind of code is, I don't want to commit > > >> any substantive changes untested and uncompiled. > > >> > > >> More below: > > >> > > >>> On 07/07/2016 04:24 AM, Jay K wrote: > > >>> So...I do NOT understand all of this. > > >>> > > >>> > > >>> However, comparing the three implementations > > >>> and attempting to understand ours, > > >>> I am struck by the following > > >>> > > >>> - Our broadcast seems ok, but > > >>> - our signal seems to wake everyone, > > >>> and then...they race a bit, > > >>> one will decide it is last, and > > >>> reset the event, but every prior waiter > > >>> is still woken. > > >>> Why not merge the following chunks: > > >> > > >> I don't think this will work. It is important that, once a thread has decided, > > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > > >> if it is also the last of its generation to proceed, resets c.waitevent. This > > >> is ensured by its having already reacquired m first, preventing any other thread > > >> from returning from its Wait yet. > > > > I was wrong about this. There is already no fairness being enforced here. Even with > > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > > > Merging, as you proposed, is probably close to the fix for this bug. > > > > > > >> > > >> And it can't attempt to acquire m while already holding conditionLock, because this would > > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > > >> already holding m, at line 265. > > >> > > >>> > > >>> > > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > > >>> . > > >>> . > > >>> . > > >>> 1 > > >>> EnterCriticalSection(conditionLock); > > >>> waitDone := (c.tickets # 0 AND c.counter # count); > > >> A:---------^ > > >> B:--------------------------------------------------^ > > >>> LeaveCriticalSection(conditionLock); > > >>> END; (* WHILE *) > > >>> IF waitDone THEN > > >>> alerted := FALSE; > > >>> END; > > >>> m.acquire(); > > >>> 2 > > >>> EnterCriticalSection(conditionLock); > > >>> DEC(c.waiters); > > >>> IF waitDone THEN esp. here. > > >>> DEC(c.tickets); > > >> > > >> C:---------------^ > > >>> lastWaiter := (c.tickets = 0); > > >>> END; > > >>> LeaveCriticalSection(conditionLock); > > >>> > > >>> > > >>> That is, if we decrement tickets earlier > > >>> within the first critical section, > > >>> while we will still wake everyone, > > >>> only one will decide waitDone and the rest will keep looping. > > >>> A downside of this, perhaps, is that waking all for Broadcast > > >>> might be a little slower. > > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > > >>> implementations, they seem to deal with this differently than us, > > >>> and they each do about the same thing -- they use a counted semaphore. > > >>> In the boost case, it appears they duplicate the semaphore for every > > >>> notification generation, which seems expensive. > > >>> > > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > > >>> Or they can do their own reference counting? > > >>> jdk7 seems to looke like jdk6. > > >>> The code is gone in jdk8 and I can't easily find the delete in history. > > >>> The lock merging jdk does probably helps here too. > > >>> In fact they merge #1 and #2 above as a result. > > >>> But they still initially wake all threads for signal, not just broadcast. > > >>> There is also the problem that all the event waits > > >>> are followed by EnterCriticalSection (or jdk facsimile). > > >>> - Jay > > >>> > > >>> > > >>> > > >>> > > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > --! > > >> --- > > >>> From: jay.krell at cornell.edu > > >>> To: m3devel at elegosoft.com > > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > > >>> > > >>> Here is another implementation to consider: > > >>> > > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > > >>> > > >>> - Jay > > >>> > > >>> > > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > --! > > >> --- > > >>> From: jay.krell at cornell.edu > > >>> To: m3devel at elegosoft.com > > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > > >>> Subject: [M3devel] purported condition variable problems on Win32? > > >>> > > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > > >>> > > >>> vs. > > >>> > > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > >>> vs. > > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > > >>> I wrote this: > > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > > >>> alertable: BOOLEAN) RAISES {Alerted} = > > >>> (* LL = m on entry and exit, but not for the duration > > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > > >>> * NOTE that they merge the user lock and the condition lock. > > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > >>> * "3.3. The Generation Count Solution" > > >>> *) > > >>> > > >>> Do we have the problems described in README.CV? > > >>> > > >>> I haven't looked through the ACE code to see > > >>> to what extent they resemble solution 3.3, or if they > > >>> changed as a result of this discussion -- which I admit I don't understand > > >>> and haven't read closely. > > >>> > > >>> > > >>> Spurious wakeups are ok, though should be minimized. > > >>> > > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > >>> > > >>> Thank you, > > >>> - Jay > > >>> > > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > >>> > > >>> > > >>> _______________________________________________ > > >>> M3devel mailing list > > >>> M3devel at elegosoft.com > > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > >> > > >> -- > > >> Rodney Bates > > >> rodney.m.bates at acm.org > > >> > > >> > > > > > > > -- > > Rodney Bates > > rodney.m.bates at acm.org -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Fri Jul 22 06:24:45 2016 From: jay.krell at cornell.edu (Jay) Date: Fri, 22 Jul 2016 04:24:45 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: <57910079.6090705@lcwb.coop> References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop> <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> <578FAA2F.6020105@lcwb.coop> <57910079.6090705@lcwb.coop> Message-ID: <0C7BFA25-E353-4E89-A739-C33CF5290A6E@gmail.com> Do our semantics allow merging the locks? I.e. Must always use same lock with same condition and vice versa? That would indeed be a nice optimization, if the interface is limited that way. I'll look around later... - Jay > On Jul 21, 2016, at 10:03 AM, "Rodney M. Bates" wrote: > > I think the first lost-wakeup bug can be fixed by merging the two lock sections > on c.lock or maybe moving just the DEC(c.tickets) up to the first. I would > need to write sample code, then review it to be confident. I have not done > that so far. > > Maybe the Java folks knew this and got it right. > >> On 07/21/2016 01:43 AM, Jay K wrote: >> That question was meant for the other bug. >> >> Can this one be fixed by rechecking if tickets is positive before decrementing it? >> This seems like a big problem in Schmidt's code. >> >> I'm nervous about merging the locks, but maybe. >> In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. >> Though I realize that is actually necessarily part of what gets done. >> Schmidt clearly did not merge the locks, and the Java version does. >> The Java version also has its own simple and not-terrible critical section. >> The Java version also disappeared in newer releases and I didn't track down what happened to it. >> >> - Jay >> >> >> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --- >> From: jay.krell at cornell.edu >> To: rodney.m.bates at acm.org >> CC: m3devel at elegosoft.com >> Subject: RE: [M3devel] purported condition variable problems on Win32? >> Date: Thu, 21 Jul 2016 06:35:41 +0000 >> >> Am I understanding this? >> There are some waiters. >> There is a broadcast ("wake all"). Some waiters are woken. >> There are more waiters. >> There is a signal ("wake one"). >> There is a chance that some of the original waiters will remain waiting while the later waiters are not. >> That is, the signaled threads kind of steal the wake intended by the broadcast. >> >> >> Is this unfairness or incorrectness? >> It feels like unfairness. >> It feels like condition variables are specified fairly weakly. >> >> >> - Jay >> >> >> > Date: Wed, 20 Jul 2016 11:43:27 -0500 >> > From: rodney_bates at lcwb.coop >> > To: jay.krell at cornell.edu; rodney.m.bates at acm.org >> > CC: m3devel at elegosoft.com >> > Subject: Re: [M3devel] purported condition variable problems on Win32? >> > >> > In looking at ThreadWin32.m3 more, I think I see a significant bug. >> > >> > This gets confusing, since there are Modula-3 waits, signals, >> > etc. and similar concepts in Win32, but which are not the >> > same. I am going the try to put "M3-" and "Win-" prefixes on >> > things to make this clearer. I cartainly need to do this >> > for my own benefit, if for nobody else. >> > >> > Suppose two threads are both M3-waiting in Wait(m,c,...), further >> > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. >> > c.tickets = 0, and c.waiters = 2. >> > >> > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), >> > which will allow both the waiters to Win-wakeup sometime soon. >> > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly >> > to eventually allow only one waiter to M3-wakeup) and increments >> > c.counter, setting things up so that the waitDone expression at :308 >> > will be TRUE for the waiting threads, the next time they evaluate it. >> > >> > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, >> > releases c.lock, and tries to acquire m. The problem arises >> > because the second waiter can get to these steps before the first >> > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The >> > second to reach :308 will find c.tickets # 0, as did the first, >> > and proceed. >> > >> > Each will eventually, do a M3-wake. If this were Posix condition >> > variables, this would be wrong, because Posix says only one >> > waiter proceeds. For M3, it's OK, if unnecessary, because M3 >> > says one or more waiters proceed. However, each will, in the >> > process, get to :322, and DEC(c.tickets). Signal only provided >> > one ticket, but the waiters have taken two. c.tickets goes >> > negative. >> > >> > After this, when some thread can do another (M3-)Wait, and some >> > other does another (M3-)Signal, c.tickets increments only back to >> > zero, which means the new waiter will not M3-wake, even though it >> > should. >> > >> > >> > I am going to try to construct a test case that will force >> > this scenario. >> > >> > See more below: >> > >> > >> > On 07/18/2016 08:53 PM, Jay wrote: >> > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. >> > > >> > > Otherwise previous version I think had a very large lock. >> > > >> > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. >> > > >> > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. >> > > >> > > I guess you are saying it would be unfair, which is ok. >> > > >> > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. >> > > >> > > I'll check on the unused event. >> > > >> > > - Jay >> > > >> > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> I spent some time looking over the signal part of ThreadWin32.m3. A few >> > >> comments: >> > >> >> > >> This looks like a very direct implementation of Schmidts' generation count >> > >> algorithm, with a number of important little details additionally taken care >> > >> of. >> > >> >> > >> On small difference is at line 308 (B, below) where Schmidt does >> > >> c.counter>count instead of #. Usually, this would make no difference, >> > >> since counter only increases, but on a long-running program, it could >> > >> overflow. If this happened,as it often does, as silent wrap-around to >> > >> FIRST(INTEGER), some threads could be trapped for an extremely long >> > >> time, waiting for counter come back around greater than their count. As >> > >> coded in ThreadWin32, they would be allowed to proceed normally. >> > >> >> > >> The only glitch would be that, if there were simultaneously waiting threads >> > >> separated by NUMBER(INTEGER) truly different generations (but the same value >> > >> of c.counter), the recent ones could unfairly proceed along with the >> > >> extremely old ones. That seems extremely less likely than the >> > >> mere existence of an overflow. >> > >> >> > >> Both type Condition and type Activation have a field waitEvent: HANDLE. >> > >> Condition.waitEvent is extensively used, but Activation.waitEvent is >> > >> only created at :537 and deleted at :554, but never used in any real >> > >> way. It could be deleted. >> > >> >> > >> As for this code: >> > >> >> > >> EnterCriticalSection(conditionLock); >> > >> >> > >> (* Capture the value of the counter before we start waiting. >> > >> * We will not stop waiting until the counter changes. >> > >> * That is, we will not stop waiting until a signal >> > >> * comes in after we start waiting. >> > >> *) >> > >> >> > >> count := c.counter; >> > >> INC(c.waiters); >> > >> >> > >> LeaveCriticalSection(conditionLock); >> > >> m.release(); (* can this be moved to before taking conditionLock? *) >> > >> >> > >> No, it is important to sample and save c.counter in the order threads wait. >> > >> Retaining m until this is done ensures this. Otherwise, some other thread >> > >> that waited later could have altered c.counter before this one gets conditionLock >> > >> and saves its copy of c.counter. >> > >> >> > >> I have made several comment changes to ThreadWin32.m3 that would have made >> > >> it easier to vet. I will commit these, but only comments. It is either >> > >> very difficult or impossible for me to test or even recompile this module, >> > >> and, especially as fragile as this kind of code is, I don't want to commit >> > >> any substantive changes untested and uncompiled. >> > >> >> > >> More below: >> > >> >> > >>> On 07/07/2016 04:24 AM, Jay K wrote: >> > >>> So...I do NOT understand all of this. >> > >>> >> > >>> >> > >>> However, comparing the three implementations >> > >>> and attempting to understand ours, >> > >>> I am struck by the following >> > >>> >> > >>> - Our broadcast seems ok, but >> > >>> - our signal seems to wake everyone, >> > >>> and then...they race a bit, >> > >>> one will decide it is last, and >> > >>> reset the event, but every prior waiter >> > >>> is still woken. >> > >>> Why not merge the following chunks: >> > >> >> > >> I don't think this will work. It is important that, once a thread has decided, >> > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed >> > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, >> > >> if it is also the last of its generation to proceed, resets c.waitevent. This >> > >> is ensured by its having already reacquired m first, preventing any other thread >> > >> from returning from its Wait yet. >> > >> > I was wrong about this. There is already no fairness being enforced here. Even with >> > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is >> > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. >> > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters >> > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. >> > >> > Merging, as you proposed, is probably close to the fix for this bug. >> > >> > >> > >> >> > >> And it can't attempt to acquire m while already holding conditionLock, because this would >> > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while >> > >> already holding m, at line 265. >> > >> >> > >>> >> > >>> >> > >>> WHILE (NOT alerted) AND (NOT waitDone) DO >> > >>> . >> > >>> . >> > >>> . >> > >>> 1 >> > >>> EnterCriticalSection(conditionLock); >> > >>> waitDone := (c.tickets # 0 AND c.counter # count); >> > >> A:---------^ >> > >> B:--------------------------------------------------^ >> > >>> LeaveCriticalSection(conditionLock); >> > >>> END; (* WHILE *) >> > >>> IF waitDone THEN >> > >>> alerted := FALSE; >> > >>> END; >> > >>> m.acquire(); >> > >>> 2 >> > >>> EnterCriticalSection(conditionLock); >> > >>> DEC(c.waiters); >> > >>> IF waitDone THEN esp. here. >> > >>> DEC(c.tickets); >> > >> >> > >> C:---------------^ >> > >>> lastWaiter := (c.tickets = 0); >> > >>> END; >> > >>> LeaveCriticalSection(conditionLock); >> > >>> >> > >>> >> > >>> That is, if we decrement tickets earlier >> > >>> within the first critical section, >> > >>> while we will still wake everyone, >> > >>> only one will decide waitDone and the rest will keep looping. >> > >>> A downside of this, perhaps, is that waking all for Broadcast >> > >>> might be a little slower. >> > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads >> > >>> implementations, they seem to deal with this differently than us, >> > >>> and they each do about the same thing -- they use a counted semaphore. >> > >>> In the boost case, it appears they duplicate the semaphore for every >> > >>> notification generation, which seems expensive. >> > >>> >> > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. >> > >>> Or they can do their own reference counting? >> > >>> jdk7 seems to looke like jdk6. >> > >>> The code is gone in jdk8 and I can't easily find the delete in history. >> > >>> The lock merging jdk does probably helps here too. >> > >>> In fact they merge #1 and #2 above as a result. >> > >>> But they still initially wake all threads for signal, not just broadcast. >> > >>> There is also the problem that all the event waits >> > >>> are followed by EnterCriticalSection (or jdk facsimile). >> > >>> - Jay >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > -----! >> > --! >> > >> --- >> > >>> From: jay.krell at cornell.edu >> > >>> To: m3devel at elegosoft.com >> > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? >> > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 >> > >>> >> > >>> Here is another implementation to consider: >> > >>> >> > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp >> > >>> >> > >>> - Jay >> > >>> >> > >>> >> > >>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > -----! >> > --! >> > >> --- >> > >>> From: jay.krell at cornell.edu >> > >>> To: m3devel at elegosoft.com >> > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 >> > >>> Subject: [M3devel] purported condition variable problems on Win32? >> > >>> >> > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV >> > >>> >> > >>> vs. >> > >>> >> > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >> > >>> vs. >> > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain >> > >>> I wrote this: >> > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; >> > >>> alertable: BOOLEAN) RAISES {Alerted} = >> > >>> (* LL = m on entry and exit, but not for the duration >> > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp >> > >>> * NOTE that they merge the user lock and the condition lock. >> > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >> > >>> * "3.3. The Generation Count Solution" >> > >>> *) >> > >>> >> > >>> Do we have the problems described in README.CV? >> > >>> >> > >>> I haven't looked through the ACE code to see >> > >>> to what extent they resemble solution 3.3, or if they >> > >>> changed as a result of this discussion -- which I admit I don't understand >> > >>> and haven't read closely. >> > >>> >> > >>> >> > >>> Spurious wakeups are ok, though should be minimized. >> > >>> >> > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. >> > >>> >> > >>> Thank you, >> > >>> - Jay >> > >>> >> > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> > >>> >> > >>> >> > >>> _______________________________________________ >> > >>> M3devel mailing list >> > >>> M3devel at elegosoft.com >> > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> > >> >> > >> -- >> > >> Rodney Bates >> > >> rodney.m.bates at acm.org >> > >> >> > >> >> > > >> > >> > -- >> > Rodney Bates >> > rodney.m.bates at acm.org > > -- > Rodney Bates > rodney.m.bates at acm.org From jay.krell at cornell.edu Fri Jul 22 10:20:36 2016 From: jay.krell at cornell.edu (Jay K) Date: Fri, 22 Jul 2016 08:20:36 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop>, <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop>, , Message-ID: Here is very weak circumstantial evidence that the locks can be merged: https://msdn.microsoft.com/en-us/library/windows/desktop/ms682052(v=vs.85).aspx "It is often convenient to use more than one condition variable with the same lock.For example, an implementation of a reader/writer lock might use a single criticalsection but separate condition variables for readers and writers." i.e. the mapping isn't one lock to one condition, but the convenientconstruct is multiple conditions to one lock, not multiple locks to one condition. http://linux.die.net/man/3/pthread_cond_wait "The effect of using more than one mutex for concurrent pthread_cond_timedwait() orpthread_cond_wait() operations on the same condition variable is undefined; that is,a condition variable becomes bound to a unique mutex when a thread waits on the conditionvariable, and this (dynamic) binding shall end when the wait returns." i.e. while the lock used with a condition variable can change through time,there can be only one at a time. And then, presumably win32 condition variables and posix condition variablesand Modula-3 Thread.Condition are all kind of similar. Hm, ok, a problem with merging the locks, is it leaves no lock for Signal/Broadcast.Perhaps they should use atomic/Interlocked. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org CC: m3devel at elegosoft.com Subject: RE: [M3devel] purported condition variable problems on Win32? Date: Thu, 21 Jul 2016 06:43:16 +0000 That question was meant for the other bug. Can this one be fixed by rechecking if tickets is positive before decrementing it? This seems like a big problem in Schmidt's code. I'm nervous about merging the locks, but maybe. In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. Though I realize that is actually necessarily part of what gets done. Schmidt clearly did not merge the locks, and the Java version does. The Java version also has its own simple and not-terrible critical section. The Java version also disappeared in newer releases and I didn't track down what happened to it. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org CC: m3devel at elegosoft.com Subject: RE: [M3devel] purported condition variable problems on Win32? Date: Thu, 21 Jul 2016 06:35:41 +0000 Am I understanding this? There are some waiters. There is a broadcast ("wake all"). Some waiters are woken. There are more waiters. There is a signal ("wake one"). There is a chance that some of the original waiters will remain waiting while the later waiters are not. That is, the signaled threads kind of steal the wake intended by the broadcast. Is this unfairness or incorrectness? It feels like unfairness. It feels like condition variables are specified fairly weakly. - Jay > Date: Wed, 20 Jul 2016 11:43:27 -0500 > From: rodney_bates at lcwb.coop > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] purported condition variable problems on Win32? > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > This gets confusing, since there are Modula-3 waits, signals, > etc. and similar concepts in Win32, but which are not the > same. I am going the try to put "M3-" and "Win-" prefixes on > things to make this clearer. I cartainly need to do this > for my own benefit, if for nobody else. > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > c.tickets = 0, and c.waiters = 2. > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > which will allow both the waiters to Win-wakeup sometime soon. > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > to eventually allow only one waiter to M3-wakeup) and increments > c.counter, setting things up so that the waitDone expression at :308 > will be TRUE for the waiting threads, the next time they evaluate it. > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > releases c.lock, and tries to acquire m. The problem arises > because the second waiter can get to these steps before the first > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > second to reach :308 will find c.tickets # 0, as did the first, > and proceed. > > Each will eventually, do a M3-wake. If this were Posix condition > variables, this would be wrong, because Posix says only one > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > says one or more waiters proceed. However, each will, in the > process, get to :322, and DEC(c.tickets). Signal only provided > one ticket, but the waiters have taken two. c.tickets goes > negative. > > After this, when some thread can do another (M3-)Wait, and some > other does another (M3-)Signal, c.tickets increments only back to > zero, which means the new waiter will not M3-wake, even though it > should. > > > I am going to try to construct a test case that will force > this scenario. > > See more below: > > > On 07/18/2016 08:53 PM, Jay wrote: > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > Otherwise previous version I think had a very large lock. > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > I guess you are saying it would be unfair, which is ok. > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > I'll check on the unused event. > > > > - Jay > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > >> > >> > >> > >> > >> > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > >> comments: > >> > >> This looks like a very direct implementation of Schmidts' generation count > >> algorithm, with a number of important little details additionally taken care > >> of. > >> > >> On small difference is at line 308 (B, below) where Schmidt does > >> c.counter>count instead of #. Usually, this would make no difference, > >> since counter only increases, but on a long-running program, it could > >> overflow. If this happened,as it often does, as silent wrap-around to > >> FIRST(INTEGER), some threads could be trapped for an extremely long > >> time, waiting for counter come back around greater than their count. As > >> coded in ThreadWin32, they would be allowed to proceed normally. > >> > >> The only glitch would be that, if there were simultaneously waiting threads > >> separated by NUMBER(INTEGER) truly different generations (but the same value > >> of c.counter), the recent ones could unfairly proceed along with the > >> extremely old ones. That seems extremely less likely than the > >> mere existence of an overflow. > >> > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > >> only created at :537 and deleted at :554, but never used in any real > >> way. It could be deleted. > >> > >> As for this code: > >> > >> EnterCriticalSection(conditionLock); > >> > >> (* Capture the value of the counter before we start waiting. > >> * We will not stop waiting until the counter changes. > >> * That is, we will not stop waiting until a signal > >> * comes in after we start waiting. > >> *) > >> > >> count := c.counter; > >> INC(c.waiters); > >> > >> LeaveCriticalSection(conditionLock); > >> m.release(); (* can this be moved to before taking conditionLock? *) > >> > >> No, it is important to sample and save c.counter in the order threads wait. > >> Retaining m until this is done ensures this. Otherwise, some other thread > >> that waited later could have altered c.counter before this one gets conditionLock > >> and saves its copy of c.counter. > >> > >> I have made several comment changes to ThreadWin32.m3 that would have made > >> it easier to vet. I will commit these, but only comments. It is either > >> very difficult or impossible for me to test or even recompile this module, > >> and, especially as fragile as this kind of code is, I don't want to commit > >> any substantive changes untested and uncompiled. > >> > >> More below: > >> > >>> On 07/07/2016 04:24 AM, Jay K wrote: > >>> So...I do NOT understand all of this. > >>> > >>> > >>> However, comparing the three implementations > >>> and attempting to understand ours, > >>> I am struck by the following > >>> > >>> - Our broadcast seems ok, but > >>> - our signal seems to wake everyone, > >>> and then...they race a bit, > >>> one will decide it is last, and > >>> reset the event, but every prior waiter > >>> is still woken. > >>> Why not merge the following chunks: > >> > >> I don't think this will work. It is important that, once a thread has decided, > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > >> if it is also the last of its generation to proceed, resets c.waitevent. This > >> is ensured by its having already reacquired m first, preventing any other thread > >> from returning from its Wait yet. > > I was wrong about this. There is already no fairness being enforced here. Even with > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > Merging, as you proposed, is probably close to the fix for this bug. > > > >> > >> And it can't attempt to acquire m while already holding conditionLock, because this would > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > >> already holding m, at line 265. > >> > >>> > >>> > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > >>> . > >>> . > >>> . > >>> 1 > >>> EnterCriticalSection(conditionLock); > >>> waitDone := (c.tickets # 0 AND c.counter # count); > >> A:---------^ > >> B:--------------------------------------------------^ > >>> LeaveCriticalSection(conditionLock); > >>> END; (* WHILE *) > >>> IF waitDone THEN > >>> alerted := FALSE; > >>> END; > >>> m.acquire(); > >>> 2 > >>> EnterCriticalSection(conditionLock); > >>> DEC(c.waiters); > >>> IF waitDone THEN esp. here. > >>> DEC(c.tickets); > >> > >> C:---------------^ > >>> lastWaiter := (c.tickets = 0); > >>> END; > >>> LeaveCriticalSection(conditionLock); > >>> > >>> > >>> That is, if we decrement tickets earlier > >>> within the first critical section, > >>> while we will still wake everyone, > >>> only one will decide waitDone and the rest will keep looping. > >>> A downside of this, perhaps, is that waking all for Broadcast > >>> might be a little slower. > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > >>> implementations, they seem to deal with this differently than us, > >>> and they each do about the same thing -- they use a counted semaphore. > >>> In the boost case, it appears they duplicate the semaphore for every > >>> notification generation, which seems expensive. > >>> > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > >>> Or they can do their own reference counting? > >>> jdk7 seems to looke like jdk6. > >>> The code is gone in jdk8 and I can't easily find the delete in history. > >>> The lock merging jdk does probably helps here too. > >>> In fact they merge #1 and #2 above as a result. > >>> But they still initially wake all threads for signal, not just broadcast. > >>> There is also the problem that all the event waits > >>> are followed by EnterCriticalSection (or jdk facsimile). > >>> - Jay > >>> > >>> > >>> > >>> > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! > >> --- > >>> From: jay.krell at cornell.edu > >>> To: m3devel at elegosoft.com > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > >>> > >>> Here is another implementation to consider: > >>> > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > >>> > >>> - Jay > >>> > >>> > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! > >> --- > >>> From: jay.krell at cornell.edu > >>> To: m3devel at elegosoft.com > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > >>> Subject: [M3devel] purported condition variable problems on Win32? > >>> > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > >>> > >>> vs. > >>> > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >>> vs. > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > >>> I wrote this: > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > >>> alertable: BOOLEAN) RAISES {Alerted} = > >>> (* LL = m on entry and exit, but not for the duration > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > >>> * NOTE that they merge the user lock and the condition lock. > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >>> * "3.3. The Generation Count Solution" > >>> *) > >>> > >>> Do we have the problems described in README.CV? > >>> > >>> I haven't looked through the ACE code to see > >>> to what extent they resemble solution 3.3, or if they > >>> changed as a result of this discussion -- which I admit I don't understand > >>> and haven't read closely. > >>> > >>> > >>> Spurious wakeups are ok, though should be minimized. > >>> > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > >>> > >>> Thank you, > >>> - Jay > >>> > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >>> > >>> > >>> _______________________________________________ > >>> M3devel mailing list > >>> M3devel at elegosoft.com > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >> > >> -- > >> Rodney Bates > >> rodney.m.bates at acm.org > >> > >> > > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Jul 22 10:32:27 2016 From: jay.krell at cornell.edu (Jay K) Date: Fri, 22 Jul 2016 08:32:27 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop>, <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop>, , , Message-ID: It works in Java because: - "mutex" and "condition" are merged into "monitor" - callers of notify/notifyAll are required to be holding the monitor's lock - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org CC: m3devel at elegosoft.com Subject: RE: [M3devel] purported condition variable problems on Win32? Date: Fri, 22 Jul 2016 08:20:32 +0000 Here is very weak circumstantial evidence that the locks can be merged: https://msdn.microsoft.com/en-us/library/windows/desktop/ms682052(v=vs.85).aspx "It is often convenient to use more than one condition variable with the same lock.For example, an implementation of a reader/writer lock might use a single criticalsection but separate condition variables for readers and writers." i.e. the mapping isn't one lock to one condition, but the convenientconstruct is multiple conditions to one lock, not multiple locks to one condition. http://linux.die.net/man/3/pthread_cond_wait "The effect of using more than one mutex for concurrent pthread_cond_timedwait() orpthread_cond_wait() operations on the same condition variable is undefined; that is,a condition variable becomes bound to a unique mutex when a thread waits on the conditionvariable, and this (dynamic) binding shall end when the wait returns." i.e. while the lock used with a condition variable can change through time,there can be only one at a time. And then, presumably win32 condition variables and posix condition variablesand Modula-3 Thread.Condition are all kind of similar. Hm, ok, a problem with merging the locks, is it leaves no lock for Signal/Broadcast.Perhaps they should use atomic/Interlocked. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org CC: m3devel at elegosoft.com Subject: RE: [M3devel] purported condition variable problems on Win32? Date: Thu, 21 Jul 2016 06:43:16 +0000 That question was meant for the other bug. Can this one be fixed by rechecking if tickets is positive before decrementing it? This seems like a big problem in Schmidt's code. I'm nervous about merging the locks, but maybe. In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. Though I realize that is actually necessarily part of what gets done. Schmidt clearly did not merge the locks, and the Java version does. The Java version also has its own simple and not-terrible critical section. The Java version also disappeared in newer releases and I didn't track down what happened to it. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org CC: m3devel at elegosoft.com Subject: RE: [M3devel] purported condition variable problems on Win32? Date: Thu, 21 Jul 2016 06:35:41 +0000 Am I understanding this? There are some waiters. There is a broadcast ("wake all"). Some waiters are woken. There are more waiters. There is a signal ("wake one"). There is a chance that some of the original waiters will remain waiting while the later waiters are not. That is, the signaled threads kind of steal the wake intended by the broadcast. Is this unfairness or incorrectness? It feels like unfairness. It feels like condition variables are specified fairly weakly. - Jay > Date: Wed, 20 Jul 2016 11:43:27 -0500 > From: rodney_bates at lcwb.coop > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] purported condition variable problems on Win32? > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > This gets confusing, since there are Modula-3 waits, signals, > etc. and similar concepts in Win32, but which are not the > same. I am going the try to put "M3-" and "Win-" prefixes on > things to make this clearer. I cartainly need to do this > for my own benefit, if for nobody else. > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > c.tickets = 0, and c.waiters = 2. > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > which will allow both the waiters to Win-wakeup sometime soon. > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > to eventually allow only one waiter to M3-wakeup) and increments > c.counter, setting things up so that the waitDone expression at :308 > will be TRUE for the waiting threads, the next time they evaluate it. > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > releases c.lock, and tries to acquire m. The problem arises > because the second waiter can get to these steps before the first > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > second to reach :308 will find c.tickets # 0, as did the first, > and proceed. > > Each will eventually, do a M3-wake. If this were Posix condition > variables, this would be wrong, because Posix says only one > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > says one or more waiters proceed. However, each will, in the > process, get to :322, and DEC(c.tickets). Signal only provided > one ticket, but the waiters have taken two. c.tickets goes > negative. > > After this, when some thread can do another (M3-)Wait, and some > other does another (M3-)Signal, c.tickets increments only back to > zero, which means the new waiter will not M3-wake, even though it > should. > > > I am going to try to construct a test case that will force > this scenario. > > See more below: > > > On 07/18/2016 08:53 PM, Jay wrote: > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > Otherwise previous version I think had a very large lock. > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > I guess you are saying it would be unfair, which is ok. > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > I'll check on the unused event. > > > > - Jay > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > >> > >> > >> > >> > >> > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > >> comments: > >> > >> This looks like a very direct implementation of Schmidts' generation count > >> algorithm, with a number of important little details additionally taken care > >> of. > >> > >> On small difference is at line 308 (B, below) where Schmidt does > >> c.counter>count instead of #. Usually, this would make no difference, > >> since counter only increases, but on a long-running program, it could > >> overflow. If this happened,as it often does, as silent wrap-around to > >> FIRST(INTEGER), some threads could be trapped for an extremely long > >> time, waiting for counter come back around greater than their count. As > >> coded in ThreadWin32, they would be allowed to proceed normally. > >> > >> The only glitch would be that, if there were simultaneously waiting threads > >> separated by NUMBER(INTEGER) truly different generations (but the same value > >> of c.counter), the recent ones could unfairly proceed along with the > >> extremely old ones. That seems extremely less likely than the > >> mere existence of an overflow. > >> > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > >> only created at :537 and deleted at :554, but never used in any real > >> way. It could be deleted. > >> > >> As for this code: > >> > >> EnterCriticalSection(conditionLock); > >> > >> (* Capture the value of the counter before we start waiting. > >> * We will not stop waiting until the counter changes. > >> * That is, we will not stop waiting until a signal > >> * comes in after we start waiting. > >> *) > >> > >> count := c.counter; > >> INC(c.waiters); > >> > >> LeaveCriticalSection(conditionLock); > >> m.release(); (* can this be moved to before taking conditionLock? *) > >> > >> No, it is important to sample and save c.counter in the order threads wait. > >> Retaining m until this is done ensures this. Otherwise, some other thread > >> that waited later could have altered c.counter before this one gets conditionLock > >> and saves its copy of c.counter. > >> > >> I have made several comment changes to ThreadWin32.m3 that would have made > >> it easier to vet. I will commit these, but only comments. It is either > >> very difficult or impossible for me to test or even recompile this module, > >> and, especially as fragile as this kind of code is, I don't want to commit > >> any substantive changes untested and uncompiled. > >> > >> More below: > >> > >>> On 07/07/2016 04:24 AM, Jay K wrote: > >>> So...I do NOT understand all of this. > >>> > >>> > >>> However, comparing the three implementations > >>> and attempting to understand ours, > >>> I am struck by the following > >>> > >>> - Our broadcast seems ok, but > >>> - our signal seems to wake everyone, > >>> and then...they race a bit, > >>> one will decide it is last, and > >>> reset the event, but every prior waiter > >>> is still woken. > >>> Why not merge the following chunks: > >> > >> I don't think this will work. It is important that, once a thread has decided, > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > >> if it is also the last of its generation to proceed, resets c.waitevent. This > >> is ensured by its having already reacquired m first, preventing any other thread > >> from returning from its Wait yet. > > I was wrong about this. There is already no fairness being enforced here. Even with > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > Merging, as you proposed, is probably close to the fix for this bug. > > > >> > >> And it can't attempt to acquire m while already holding conditionLock, because this would > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > >> already holding m, at line 265. > >> > >>> > >>> > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > >>> . > >>> . > >>> . > >>> 1 > >>> EnterCriticalSection(conditionLock); > >>> waitDone := (c.tickets # 0 AND c.counter # count); > >> A:---------^ > >> B:--------------------------------------------------^ > >>> LeaveCriticalSection(conditionLock); > >>> END; (* WHILE *) > >>> IF waitDone THEN > >>> alerted := FALSE; > >>> END; > >>> m.acquire(); > >>> 2 > >>> EnterCriticalSection(conditionLock); > >>> DEC(c.waiters); > >>> IF waitDone THEN esp. here. > >>> DEC(c.tickets); > >> > >> C:---------------^ > >>> lastWaiter := (c.tickets = 0); > >>> END; > >>> LeaveCriticalSection(conditionLock); > >>> > >>> > >>> That is, if we decrement tickets earlier > >>> within the first critical section, > >>> while we will still wake everyone, > >>> only one will decide waitDone and the rest will keep looping. > >>> A downside of this, perhaps, is that waking all for Broadcast > >>> might be a little slower. > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > >>> implementations, they seem to deal with this differently than us, > >>> and they each do about the same thing -- they use a counted semaphore. > >>> In the boost case, it appears they duplicate the semaphore for every > >>> notification generation, which seems expensive. > >>> > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > >>> Or they can do their own reference counting? > >>> jdk7 seems to looke like jdk6. > >>> The code is gone in jdk8 and I can't easily find the delete in history. > >>> The lock merging jdk does probably helps here too. > >>> In fact they merge #1 and #2 above as a result. > >>> But they still initially wake all threads for signal, not just broadcast. > >>> There is also the problem that all the event waits > >>> are followed by EnterCriticalSection (or jdk facsimile). > >>> - Jay > >>> > >>> > >>> > >>> > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! > >> --- > >>> From: jay.krell at cornell.edu > >>> To: m3devel at elegosoft.com > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > >>> > >>> Here is another implementation to consider: > >>> > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > >>> > >>> - Jay > >>> > >>> > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! > >> --- > >>> From: jay.krell at cornell.edu > >>> To: m3devel at elegosoft.com > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > >>> Subject: [M3devel] purported condition variable problems on Win32? > >>> > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > >>> > >>> vs. > >>> > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >>> vs. > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > >>> I wrote this: > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > >>> alertable: BOOLEAN) RAISES {Alerted} = > >>> (* LL = m on entry and exit, but not for the duration > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > >>> * NOTE that they merge the user lock and the condition lock. > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >>> * "3.3. The Generation Count Solution" > >>> *) > >>> > >>> Do we have the problems described in README.CV? > >>> > >>> I haven't looked through the ACE code to see > >>> to what extent they resemble solution 3.3, or if they > >>> changed as a result of this discussion -- which I admit I don't understand > >>> and haven't read closely. > >>> > >>> > >>> Spurious wakeups are ok, though should be minimized. > >>> > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > >>> > >>> Thank you, > >>> - Jay > >>> > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >>> > >>> > >>> _______________________________________________ > >>> M3devel mailing list > >>> M3devel at elegosoft.com > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >> > >> -- > >> Rodney Bates > >> rodney.m.bates at acm.org > >> > >> > > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Sat Jul 23 23:02:15 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sat, 23 Jul 2016 21:02:15 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop>, <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop>, , Message-ID: <5793DB37.10603@lcwb.coop> On 07/22/2016 03:20 AM, Jay K wrote: > Here is very weak circumstantial evidence that the locks can be merged: > > > > https://msdn.microsoft.com/en-us/library/windows/desktop/ms682052(v=vs.85).aspx > > "It is often convenient to use more than one condition variable with the same lock. > For example, an implementation of a reader/writer lock might use a single critical > section but separate condition variables for readers and writers. > " > > > i.e. the mapping isn't one lock to one condition, but the convenient > construct is multiple conditions to one lock, not multiple locks to one condition. > > > http://linux.die.net/man/3/pthread_cond_wait > > " > The effect of using more than one mutex for concurrent pthread_cond_timedwait() or > pthread_cond_wait() operations on the same condition variable is undefined; that is, > a condition variable becomes bound to a unique mutex when a thread waits on the condition > variable, and this (dynamic) binding shall end when the wait returns. > " > Yes, these are the usual rules about the relationship between mutexs and condition variables. There is sometimes a definite need for >1 condition variable associated with a single mutex. The other way around might be definable, but I think would create a big mess. Our Thread interface documents only that Wait(m,c) must have m locked at the time of call (and will return with it locked too, but not necessarily for the duration. Chapter 4 of SPwM3, "An Introduction to Programming with Threads" informally defines a _monitor_ as "some data, a mutex, and zero or more condition variables" and says further "A particular condition variable is always used in conjunction with the same mutex and its data". (SPwM3, 4.3.2, p93) It looks to me like the formal specification of Wait, Signal, and Broadcast in SRwM3, 5.3.3, p124 would allow a condition to simultaneously contain waiters that passed in different mutexes. Each thread remembers locally what mutex it wants to reacquire when it wakes up. I'm not sure we really want to commit to that. Client code's doing it would no doubt create even trickier code than this kind of stuff already is. Off hand, I have a hard time thinking of a reasonable use-case. If we decide to go with the informal statement, the implementations really should enforce it by storing the associated mutex in the condition variable, and verifying that additional waiters supply the same one. Probably, in most implementations, you could get away with moving the association of a condition variable from one mutex to another when no threads were waiting on the condition. Buy\t the only possible benefit I can think of would be memory savings, not needing as many condition variables, since the different mutex associations would be, well, mutually exclusive. > i.e. while the lock used with a condition variable can change through time, > there can be only one at a time. > > And then, presumably win32 condition variables and posix condition variables > and Modula-3 Thread.Condition are all kind of similar. > > Hm, ok, a problem with merging the locks, is it leaves no lock for Signal/Broadcast. > Perhaps they should use atomic/Interlocked. > > - Jay > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > From: jay.krell at cornell.edu > To: rodney.m.bates at acm.org > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] purported condition variable problems on Win32? > Date: Thu, 21 Jul 2016 06:43:16 +0000 > > That question was meant for the other bug. > > Can this one be fixed by rechecking if tickets is positive before decrementing it? > This seems like a big problem in Schmidt's code. > > I'm nervous about merging the locks, but maybe. > In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. > Though I realize that is actually necessarily part of what gets done. > Schmidt clearly did not merge the locks, and the Java version does. > The Java version also has its own simple and not-terrible critical section. > The Java version also disappeared in newer releases and I didn't track down what happened to it. > > - Jay > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > From: jay.krell at cornell.edu > To: rodney.m.bates at acm.org > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] purported condition variable problems on Win32? > Date: Thu, 21 Jul 2016 06:35:41 +0000 > > Am I understanding this? > There are some waiters. > There is a broadcast ("wake all"). Some waiters are woken. > There are more waiters. > There is a signal ("wake one"). > There is a chance that some of the original waiters will remain waiting while the later waiters are not. > That is, the signaled threads kind of steal the wake intended by the broadcast. > > > Is this unfairness or incorrectness? > It feels like unfairness. > It feels like condition variables are specified fairly weakly. > > > - Jay > > > > Date: Wed, 20 Jul 2016 11:43:27 -0500 > > From: rodney_bates at lcwb.coop > > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] purported condition variable problems on Win32? > > > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > > > This gets confusing, since there are Modula-3 waits, signals, > > etc. and similar concepts in Win32, but which are not the > > same. I am going the try to put "M3-" and "Win-" prefixes on > > things to make this clearer. I cartainly need to do this > > for my own benefit, if for nobody else. > > > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > > c.tickets = 0, and c.waiters = 2. > > > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > > which will allow both the waiters to Win-wakeup sometime soon. > > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > > to eventually allow only one waiter to M3-wakeup) and increments > > c.counter, setting things up so that the waitDone expression at :308 > > will be TRUE for the waiting threads, the next time they evaluate it. > > > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > > releases c.lock, and tries to acquire m. The problem arises > > because the second waiter can get to these steps before the first > > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > > second to reach :308 will find c.tickets # 0, as did the first, > > and proceed. > > > > Each will eventually, do a M3-wake. If this were Posix condition > > variables, this would be wrong, because Posix says only one > > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > > says one or more waiters proceed. However, each will, in the > > process, get to :322, and DEC(c.tickets). Signal only provided > > one ticket, but the waiters have taken two. c.tickets goes > > negative. > > > > After this, when some thread can do another (M3-)Wait, and some > > other does another (M3-)Signal, c.tickets increments only back to > > zero, which means the new waiter will not M3-wake, even though it > > should. > > > > > > I am going to try to construct a test case that will force > > this scenario. > > > > See more below: > > > > > > On 07/18/2016 08:53 PM, Jay wrote: > > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > > > Otherwise previous version I think had a very large lock. > > > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > > > I guess you are saying it would be unfair, which is ok. > > > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > > > I'll check on the unused event. > > > > > > - Jay > > > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > > >> > > >> > > >> > > >> > > >> > > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > > >> comments: > > >> > > >> This looks like a very direct implementation of Schmidts' generation count > > >> algorithm, with a number of important little details additionally taken care > > >> of. > > >> > > >> On small difference is at line 308 (B, below) where Schmidt does > > >> c.counter>count instead of #. Usually, this would make no difference, > > >> since counter only increases, but on a long-running program, it could > > >> overflow. If this happened,as it often does, as silent wrap-around to > > >> FIRST(INTEGER), some threads could be trapped for an extremely long > > >> time, waiting for counter come back around greater than their count. As > > >> coded in ThreadWin32, they would be allowed to proceed normally. > > >> > > >> The only glitch would be that, if there were simultaneously waiting threads > > >> separated by NUMBER(INTEGER) truly different generations (but the same value > > >> of c.counter), the recent ones could unfairly proceed along with the > > >> extremely old ones. That seems extremely less likely than the > > >> mere existence of an overflow. > > >> > > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > > >> only created at :537 and deleted at :554, but never used in any real > > >> way. It could be deleted. > > >> > > >> As for this code: > > >> > > >> EnterCriticalSection(conditionLock); > > >> > > >> (* Capture the value of the counter before we start waiting. > > >> * We will not stop waiting until the counter changes. > > >> * That is, we will not stop waiting until a signal > > >> * comes in after we start waiting. > > >> *) > > >> > > >> count := c.counter; > > >> INC(c.waiters); > > >> > > >> LeaveCriticalSection(conditionLock); > > >> m.release(); (* can this be moved to before taking conditionLock? *) > > >> > > >> No, it is important to sample and save c.counter in the order threads wait. > > >> Retaining m until this is done ensures this. Otherwise, some other thread > > >> that waited later could have altered c.counter before this one gets conditionLock > > >> and saves its copy of c.counter. > > >> > > >> I have made several comment changes to ThreadWin32.m3 that would have made > > >> it easier to vet. I will commit these, but only comments. It is either > > >> very difficult or impossible for me to test or even recompile this module, > > >> and, especially as fragile as this kind of code is, I don't want to commit > > >> any substantive changes untested and uncompiled. > > >> > > >> More below: > > >> > > >>> On 07/07/2016 04:24 AM, Jay K wrote: > > >>> So...I do NOT understand all of this. > > >>> > > >>> > > >>> However, comparing the three implementations > > >>> and attempting to understand ours, > > >>> I am struck by the following > > >>> > > >>> - Our broadcast seems ok, but > > >>> - our signal seems to wake everyone, > > >>> and then...they race a bit, > > >>> one will decide it is last, and > > >>> reset the event, but every prior waiter > > >>> is still woken. > > >>> Why not merge the following chunks: > > >> > > >> I don't think this will work. It is important that, once a thread has decided, > > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > > >> if it is also the last of its generation to proceed, resets c.waitevent. This > > >> is ensured by its having already reacquired m first, preventing any other thread > > >> from returning from its Wait yet. > > > > I was wrong about this. There is already no fairness being enforced here. Even with > > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > > > Merging, as you proposed, is probably close to the fix for this bug. > > > > > > >> > > >> And it can't attempt to acquire m while already holding conditionLock, because this would > > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > > >> already holding m, at line 265. > > >> > > >>> > > >>> > > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > > >>> . > > >>> . > > >>> . > > >>> 1 > > >>> EnterCriticalSection(conditionLock); > > >>> waitDone := (c.tickets # 0 AND c.counter # count); > > >> A:---------^ > > >> B:--------------------------------------------------^ > > >>> LeaveCriticalSection(conditionLock); > > >>> END; (* WHILE *) > > >>> IF waitDone THEN > > >>> alerted := FALSE; > > >>> END; > > >>> m.acquire(); > > >>> 2 > > >>> EnterCriticalSection(conditionLock); > > >>> DEC(c.waiters); > > >>> IF waitDone THEN esp. here. > > >>> DEC(c.tickets); > > >> > > >> C:---------------^ > > >>> lastWaiter := (c.tickets = 0); > > >>> END; > > >>> LeaveCriticalSection(conditionLock); > > >>> > > >>> > > >>> That is, if we decrement tickets earlier > > >>> within the first critical section, > > >>> while we will still wake everyone, > > >>> only one will decide waitDone and the rest will keep looping. > > >>> A downside of this, perhaps, is that waking all for Broadcast > > >>> might be a little slower. > > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > > >>> implementations, they seem to deal with this differently than us, > > >>> and they each do about the same thing -- they use a counted semaphore. > > >>> In the boost case, it appears they duplicate the semaphore for every > > >>> notification generation, which seems expensive. > > >>> > > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > > >>> Or they can do their own reference counting? > > >>> jdk7 seems to looke like jdk6. > > >>> The code is gone in jdk8 and I can't easily find the delete in history. > > >>> The lock merging jdk does probably helps here too. > > >>> In fact they merge #1 and #2 above as a result. > > >>> But they still initially wake all threads for signal, not just broadcast. > > >>> There is also the problem that all the event waits > > >>> are followed by EnterCriticalSection (or jdk facsimile). > > >>> - Jay > > >>> > > >>> > > >>> > > >>> > > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > --! > > >> --- > > >>> From: jay.krell at cornell.edu > > >>> To: m3devel at elegosoft.com > > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > > >>> > > >>> Here is another implementation to consider: > > >>> > > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > > >>> > > >>> - Jay > > >>> > > >>> > > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > --! > > >> --- > > >>> From: jay.krell at cornell.edu > > >>> To: m3devel at elegosoft.com > > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > > >>> Subject: [M3devel] purported condition variable problems on Win32? > > >>> > > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > > >>> > > >>> vs. > > >>> > > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > >>> vs. > > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > > >>> I wrote this: > > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > > >>> alertable: BOOLEAN) RAISES {Alerted} = > > >>> (* LL = m on entry and exit, but not for the duration > > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > > >>> * NOTE that they merge the user lock and the condition lock. > > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > >>> * "3.3. The Generation Count Solution" > > >>> *) > > >>> > > >>> Do we have the problems described in README.CV? > > >>> > > >>> I haven't looked through the ACE code to see > > >>> to what extent they resemble solution 3.3, or if they > > >>> changed as a result of this discussion -- which I admit I don't understand > > >>> and haven't read closely. > > >>> > > >>> > > >>> Spurious wakeups are ok, though should be minimized. > > >>> > > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > >>> > > >>> Thank you, > > >>> - Jay > > >>> > > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > >>> > > >>> > > >>> _______________________________________________ > > >>> M3devel mailing list > > >>> M3devel at elegosoft.com > > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > >> > > >> -- > > >> Rodney Bates > > >> rodney.m.bates at acm.org > > >> > > >> > > > > > > > -- > > Rodney Bates > > rodney.m.bates at acm.org -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Sun Jul 24 01:37:14 2016 From: jay.krell at cornell.edu (Jay K) Date: Sat, 23 Jul 2016 23:37:14 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: <5793DB37.10603@lcwb.coop> References: <578BF33B.2030605@lcwb.coop>, <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop>, , , , <5793DB37.10603@lcwb.coop> Message-ID: I don't think Signal/Broadcast require caller be holding any lock, is a problem. - Jay > Date: Sat, 23 Jul 2016 16:01:43 -0500 > From: rodney_bates at lcwb.coop > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] purported condition variable problems on Win32? > > > > On 07/22/2016 03:20 AM, Jay K wrote: > > Here is very weak circumstantial evidence that the locks can be merged: > > > > > > > > https://msdn.microsoft.com/en-us/library/windows/desktop/ms682052(v=vs.85).aspx > > > > "It is often convenient to use more than one condition variable with the same lock. > > For example, an implementation of a reader/writer lock might use a single critical > > section but separate condition variables for readers and writers. > > " > > > > > > i.e. the mapping isn't one lock to one condition, but the convenient > > construct is multiple conditions to one lock, not multiple locks to one condition. > > > > > > http://linux.die.net/man/3/pthread_cond_wait > > > > " > > The effect of using more than one mutex for concurrent pthread_cond_timedwait() or > > pthread_cond_wait() operations on the same condition variable is undefined; that is, > > a condition variable becomes bound to a unique mutex when a thread waits on the condition > > variable, and this (dynamic) binding shall end when the wait returns. > > " > > > > Yes, these are the usual rules about the relationship between mutexs and condition variables. > There is sometimes a definite need for >1 condition variable associated with a single > mutex. The other way around might be definable, but I think would create a big mess. > > Our Thread interface documents only that Wait(m,c) must have m locked at the time of > call (and will return with it locked too, but not necessarily for the duration. > > Chapter 4 of SPwM3, "An Introduction to Programming with Threads" informally defines > a _monitor_ as "some data, a mutex, and zero or more condition variables" and says > further "A particular condition variable is always used in conjunction with the same > mutex and its data". (SPwM3, 4.3.2, p93) > > It looks to me like the formal specification of Wait, Signal, and Broadcast in > SRwM3, 5.3.3, p124 would allow a condition to simultaneously contain waiters > that passed in different mutexes. Each thread remembers locally what mutex it > wants to reacquire when it wakes up. > > I'm not sure we really want to commit to that. Client code's doing it would no doubt > create even trickier code than this kind of stuff already is. Off hand, I have a > hard time thinking of a reasonable use-case. > > If we decide to go with the informal statement, the implementations really should > enforce it by storing the associated mutex in the condition variable, and verifying > that additional waiters supply the same one. > > Probably, in most implementations, you could get away with moving the association > of a condition variable from one mutex to another when no threads were waiting on > the condition. Buy\t the only possible benefit I can think of would be memory savings, > not needing as many condition variables, since the different mutex associations would > be, well, mutually exclusive. > > > > > > i.e. while the lock used with a condition variable can change through time, > > there can be only one at a time. > > > > And then, presumably win32 condition variables and posix condition variables > > and Modula-3 Thread.Condition are all kind of similar. > > > > Hm, ok, a problem with merging the locks, is it leaves no lock for Signal/Broadcast. > > Perhaps they should use atomic/Interlocked. > > > > - Jay > > > > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --- > > From: jay.krell at cornell.edu > > To: rodney.m.bates at acm.org > > CC: m3devel at elegosoft.com > > Subject: RE: [M3devel] purported condition variable problems on Win32? > > Date: Thu, 21 Jul 2016 06:43:16 +0000 > > > > That question was meant for the other bug. > > > > Can this one be fixed by rechecking if tickets is positive before decrementing it? > > This seems like a big problem in Schmidt's code. > > > > I'm nervous about merging the locks, but maybe. > > In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. > > Though I realize that is actually necessarily part of what gets done. > > Schmidt clearly did not merge the locks, and the Java version does. > > The Java version also has its own simple and not-terrible critical section. > > The Java version also disappeared in newer releases and I didn't track down what happened to it. > > > > - Jay > > > > > > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --- > > From: jay.krell at cornell.edu > > To: rodney.m.bates at acm.org > > CC: m3devel at elegosoft.com > > Subject: RE: [M3devel] purported condition variable problems on Win32? > > Date: Thu, 21 Jul 2016 06:35:41 +0000 > > > > Am I understanding this? > > There are some waiters. > > There is a broadcast ("wake all"). Some waiters are woken. > > There are more waiters. > > There is a signal ("wake one"). > > There is a chance that some of the original waiters will remain waiting while the later waiters are not. > > That is, the signaled threads kind of steal the wake intended by the broadcast. > > > > > > Is this unfairness or incorrectness? > > It feels like unfairness. > > It feels like condition variables are specified fairly weakly. > > > > > > - Jay > > > > > > > Date: Wed, 20 Jul 2016 11:43:27 -0500 > > > From: rodney_bates at lcwb.coop > > > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > > > CC: m3devel at elegosoft.com > > > Subject: Re: [M3devel] purported condition variable problems on Win32? > > > > > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > > > > > This gets confusing, since there are Modula-3 waits, signals, > > > etc. and similar concepts in Win32, but which are not the > > > same. I am going the try to put "M3-" and "Win-" prefixes on > > > things to make this clearer. I cartainly need to do this > > > for my own benefit, if for nobody else. > > > > > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > > > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > > > c.tickets = 0, and c.waiters = 2. > > > > > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > > > which will allow both the waiters to Win-wakeup sometime soon. > > > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > > > to eventually allow only one waiter to M3-wakeup) and increments > > > c.counter, setting things up so that the waitDone expression at :308 > > > will be TRUE for the waiting threads, the next time they evaluate it. > > > > > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > > > releases c.lock, and tries to acquire m. The problem arises > > > because the second waiter can get to these steps before the first > > > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > > > second to reach :308 will find c.tickets # 0, as did the first, > > > and proceed. > > > > > > Each will eventually, do a M3-wake. If this were Posix condition > > > variables, this would be wrong, because Posix says only one > > > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > > > says one or more waiters proceed. However, each will, in the > > > process, get to :322, and DEC(c.tickets). Signal only provided > > > one ticket, but the waiters have taken two. c.tickets goes > > > negative. > > > > > > After this, when some thread can do another (M3-)Wait, and some > > > other does another (M3-)Signal, c.tickets increments only back to > > > zero, which means the new waiter will not M3-wake, even though it > > > should. > > > > > > > > > I am going to try to construct a test case that will force > > > this scenario. > > > > > > See more below: > > > > > > > > > On 07/18/2016 08:53 PM, Jay wrote: > > > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > > > > > Otherwise previous version I think had a very large lock. > > > > > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > > > > > I guess you are saying it would be unfair, which is ok. > > > > > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > > > > > I'll check on the unused event. > > > > > > > > - Jay > > > > > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > > > >> comments: > > > >> > > > >> This looks like a very direct implementation of Schmidts' generation count > > > >> algorithm, with a number of important little details additionally taken care > > > >> of. > > > >> > > > >> On small difference is at line 308 (B, below) where Schmidt does > > > >> c.counter>count instead of #. Usually, this would make no difference, > > > >> since counter only increases, but on a long-running program, it could > > > >> overflow. If this happened,as it often does, as silent wrap-around to > > > >> FIRST(INTEGER), some threads could be trapped for an extremely long > > > >> time, waiting for counter come back around greater than their count. As > > > >> coded in ThreadWin32, they would be allowed to proceed normally. > > > >> > > > >> The only glitch would be that, if there were simultaneously waiting threads > > > >> separated by NUMBER(INTEGER) truly different generations (but the same value > > > >> of c.counter), the recent ones could unfairly proceed along with the > > > >> extremely old ones. That seems extremely less likely than the > > > >> mere existence of an overflow. > > > >> > > > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > > > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > > > >> only created at :537 and deleted at :554, but never used in any real > > > >> way. It could be deleted. > > > >> > > > >> As for this code: > > > >> > > > >> EnterCriticalSection(conditionLock); > > > >> > > > >> (* Capture the value of the counter before we start waiting. > > > >> * We will not stop waiting until the counter changes. > > > >> * That is, we will not stop waiting until a signal > > > >> * comes in after we start waiting. > > > >> *) > > > >> > > > >> count := c.counter; > > > >> INC(c.waiters); > > > >> > > > >> LeaveCriticalSection(conditionLock); > > > >> m.release(); (* can this be moved to before taking conditionLock? *) > > > >> > > > >> No, it is important to sample and save c.counter in the order threads wait. > > > >> Retaining m until this is done ensures this. Otherwise, some other thread > > > >> that waited later could have altered c.counter before this one gets conditionLock > > > >> and saves its copy of c.counter. > > > >> > > > >> I have made several comment changes to ThreadWin32.m3 that would have made > > > >> it easier to vet. I will commit these, but only comments. It is either > > > >> very difficult or impossible for me to test or even recompile this module, > > > >> and, especially as fragile as this kind of code is, I don't want to commit > > > >> any substantive changes untested and uncompiled. > > > >> > > > >> More below: > > > >> > > > >>> On 07/07/2016 04:24 AM, Jay K wrote: > > > >>> So...I do NOT understand all of this. > > > >>> > > > >>> > > > >>> However, comparing the three implementations > > > >>> and attempting to understand ours, > > > >>> I am struck by the following > > > >>> > > > >>> - Our broadcast seems ok, but > > > >>> - our signal seems to wake everyone, > > > >>> and then...they race a bit, > > > >>> one will decide it is last, and > > > >>> reset the event, but every prior waiter > > > >>> is still woken. > > > >>> Why not merge the following chunks: > > > >> > > > >> I don't think this will work. It is important that, once a thread has decided, > > > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > > > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > > > >> if it is also the last of its generation to proceed, resets c.waitevent. This > > > >> is ensured by its having already reacquired m first, preventing any other thread > > > >> from returning from its Wait yet. > > > > > > I was wrong about this. There is already no fairness being enforced here. Even with > > > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > > > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > > > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > > > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > > > > > Merging, as you proposed, is probably close to the fix for this bug. > > > > > > > > > >> > > > >> And it can't attempt to acquire m while already holding conditionLock, because this would > > > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > > > >> already holding m, at line 265. > > > >> > > > >>> > > > >>> > > > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > > > >>> . > > > >>> . > > > >>> . > > > >>> 1 > > > >>> EnterCriticalSection(conditionLock); > > > >>> waitDone := (c.tickets # 0 AND c.counter # count); > > > >> A:---------^ > > > >> B:--------------------------------------------------^ > > > >>> LeaveCriticalSection(conditionLock); > > > >>> END; (* WHILE *) > > > >>> IF waitDone THEN > > > >>> alerted := FALSE; > > > >>> END; > > > >>> m.acquire(); > > > >>> 2 > > > >>> EnterCriticalSection(conditionLock); > > > >>> DEC(c.waiters); > > > >>> IF waitDone THEN esp. here. > > > >>> DEC(c.tickets); > > > >> > > > >> C:---------------^ > > > >>> lastWaiter := (c.tickets = 0); > > > >>> END; > > > >>> LeaveCriticalSection(conditionLock); > > > >>> > > > >>> > > > >>> That is, if we decrement tickets earlier > > > >>> within the first critical section, > > > >>> while we will still wake everyone, > > > >>> only one will decide waitDone and the rest will keep looping. > > > >>> A downside of this, perhaps, is that waking all for Broadcast > > > >>> might be a little slower. > > > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > > > >>> implementations, they seem to deal with this differently than us, > > > >>> and they each do about the same thing -- they use a counted semaphore. > > > >>> In the boost case, it appears they duplicate the semaphore for every > > > >>> notification generation, which seems expensive. > > > >>> > > > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > > > >>> Or they can do their own reference counting? > > > >>> jdk7 seems to looke like jdk6. > > > >>> The code is gone in jdk8 and I can't easily find the delete in history. > > > >>> The lock merging jdk does probably helps here too. > > > >>> In fact they merge #1 and #2 above as a result. > > > >>> But they still initially wake all threads for signal, not just broadcast. > > > >>> There is also the problem that all the event waits > > > >>> are followed by EnterCriticalSection (or jdk facsimile). > > > >>> - Jay > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > -----! > > > --! > > > >> --- > > > >>> From: jay.krell at cornell.edu > > > >>> To: m3devel at elegosoft.com > > > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > > > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > > > >>> > > > >>> Here is another implementation to consider: > > > >>> > > > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > > > >>> > > > >>> - Jay > > > >>> > > > >>> > > > >>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > -----! > > > --! > > > >> --- > > > >>> From: jay.krell at cornell.edu > > > >>> To: m3devel at elegosoft.com > > > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > > > >>> Subject: [M3devel] purported condition variable problems on Win32? > > > >>> > > > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > > > >>> > > > >>> vs. > > > >>> > > > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > > >>> vs. > > > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > > > >>> I wrote this: > > > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > > > >>> alertable: BOOLEAN) RAISES {Alerted} = > > > >>> (* LL = m on entry and exit, but not for the duration > > > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > > > >>> * NOTE that they merge the user lock and the condition lock. > > > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > > >>> * "3.3. The Generation Count Solution" > > > >>> *) > > > >>> > > > >>> Do we have the problems described in README.CV? > > > >>> > > > >>> I haven't looked through the ACE code to see > > > >>> to what extent they resemble solution 3.3, or if they > > > >>> changed as a result of this discussion -- which I admit I don't understand > > > >>> and haven't read closely. > > > >>> > > > >>> > > > >>> Spurious wakeups are ok, though should be minimized. > > > >>> > > > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > > >>> > > > >>> Thank you, > > > >>> - Jay > > > >>> > > > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > > >>> > > > >>> > > > >>> _______________________________________________ > > > >>> M3devel mailing list > > > >>> M3devel at elegosoft.com > > > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > > >> > > > >> -- > > > >> Rodney Bates > > > >> rodney.m.bates at acm.org > > > >> > > > >> > > > > > > > > > > -- > > > Rodney Bates > > > rodney.m.bates at acm.org > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Sun Jul 24 18:29:34 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 24 Jul 2016 16:29:34 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop>, <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop>, , , , <5793DB37.10603@lcwb.coop> Message-ID: <5794ECAE.5070308@lcwb.coop> No, they don't. But I don't think that presents a problem. On 07/23/2016 06:37 PM, Jay K wrote: > I don't think Signal/Broadcast require caller be holding any lock, is a problem. > > - Jay > > > > > Date: Sat, 23 Jul 2016 16:01:43 -0500 > > From: rodney_bates at lcwb.coop > > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] purported condition variable problems on Win32? > > > > > > > > On 07/22/2016 03:20 AM, Jay K wrote: > > > Here is very weak circumstantial evidence that the locks can be merged: > > > > > > > > > > > > https://msdn.microsoft.com/en-us/library/windows/desktop/ms682052(v=vs.85).aspx > > > > > > "It is often convenient to use more than one condition variable with the same lock. > > > For example, an implementation of a reader/writer lock might use a single critical > > > section but separate condition variables for readers and writers. > > > " > > > > > > > > > i.e. the mapping isn't one lock to one condition, but the convenient > > > construct is multiple conditions to one lock, not multiple locks to one condition. > > > > > > > > > http://linux.die.net/man/3/pthread_cond_wait > > > > > > " > > > The effect of using more than one mutex for concurrent pthread_cond_timedwait() or > > > pthread_cond_wait() operations on the same condition variable is undefined; that is, > > > a condition variable becomes bound to a unique mutex when a thread waits on the condition > > > variable, and this (dynamic) binding shall end when the wait returns. > > > " > > > > > > > Yes, these are the usual rules about the relationship between mutexs and condition variables. > > There is sometimes a definite need for >1 condition variable associated with a single > > mutex. The other way around might be definable, but I think would create a big mess. > > > > Our Thread interface documents only that Wait(m,c) must have m locked at the time of > > call (and will return with it locked too, but not necessarily for the duration. > > > > Chapter 4 of SPwM3, "An Introduction to Programming with Threads" informally defines > > a _monitor_ as "some data, a mutex, and zero or more condition variables" and says > > further "A particular condition variable is always used in conjunction with the same > > mutex and its data". (SPwM3, 4.3.2, p93) > > > > It looks to me like the formal specification of Wait, Signal, and Broadcast in > > SRwM3, 5.3.3, p124 would allow a condition to simultaneously contain waiters > > that passed in different mutexes. Each thread remembers locally what mutex it > > wants to reacquire when it wakes up. > > > > I'm not sure we really want to commit to that. Client code's doing it would no doubt > > create even trickier code than this kind of stuff already is. Off hand, I have a > > hard time thinking of a reasonable use-case. > > > > If we decide to go with the informal statement, the implementations really should > > enforce it by storing the associated mutex in the condition variable, and verifying > > that additional waiters supply the same one. > > > > Probably, in most implementations, you could get away with moving the association > > of a condition variable from one mutex to another when no threads were waiting on > > the condition. Buy\t the only possible benefit I can think of would be memory savings, > > not needing as many condition variables, since the different mutex associations would > > be, well, mutually exclusive. > > > > > > > > > > > i.e. while the lock used with a condition variable can change through time, > > > there can be only one at a time. > > > > > > And then, presumably win32 condition variables and posix condition variables > > > and Modula-3 Thread.Condition are all kind of similar. > > > > > > Hm, ok, a problem with merging the locks, is it leaves no lock for Signal/Broadcast. > > > Perhaps they should use atomic/Interlocked. > > > > > > - Jay > > > > > > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > --- > > > From: jay.krell at cornell.edu > > > To: rodney.m.bates at acm.org > > > CC: m3devel at elegosoft.com > > > Subject: RE: [M3devel] purported condition variable problems on Win32? > > > Date: Thu, 21 Jul 2016 06:43:16 +0000 > > > > > > That question was meant for the other bug. > > > > > > Can this one be fixed by rechecking if tickets is positive before decrementing it? > > > This seems like a big problem in Schmidt's code. > > > > > > I'm nervous about merging the locks, but maybe. > > > In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. > > > Though I realize that is actually necessarily part of what gets done. > > > Schmidt clearly did not merge the locks, and the Java version does. > > > The Java version also has its own simple and not-terrible critical section. > > > The Java version also disappeared in newer releases and I didn't track down what happened to it. > > > > > > - Jay > > > > > > > > > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > --- > > > From: jay.krell at cornell.edu > > > To: rodney.m.bates at acm.org > > > CC: m3devel at elegosoft.com > > > Subject: RE: [M3devel] purported condition variable problems on Win32? > > > Date: Thu, 21 Jul 2016 06:35:41 +0000 > > > > > > Am I understanding this? > > > There are some waiters. > > > There is a broadcast ("wake all"). Some waiters are woken. > > > There are more waiters. > > > There is a signal ("wake one"). > > > There is a chance that some of the original waiters will remain waiting while the later waiters are not. > > > That is, the signaled threads kind of steal the wake intended by the broadcast. > > > > > > > > > Is this unfairness or incorrectness? > > > It feels like unfairness. > > > It feels like condition variables are specified fairly weakly. > > > > > > > > > - Jay > > > > > > > > > > Date: Wed, 20 Jul 2016 11:43:27 -0500 > > > > From: rodney_bates at lcwb.coop > > > > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > > > > CC: m3devel at elegosoft.com > > > > Subject: Re: [M3devel] purported condition variable problems on Win32? > > > > > > > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > > > > > > > This gets confusing, since there are Modula-3 waits, signals, > > > > etc. and similar concepts in Win32, but which are not the > > > > same. I am going the try to put "M3-" and "Win-" prefixes on > > > > things to make this clearer. I cartainly need to do this > > > > for my own benefit, if for nobody else. > > > > > > > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > > > > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > > > > c.tickets = 0, and c.waiters = 2. > > > > > > > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > > > > which will allow both the waiters to Win-wakeup sometime soon. > > > > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > > > > to eventually allow only one waiter to M3-wakeup) and increments > > > > c.counter, setting things up so that the waitDone expression at :308 > > > > will be TRUE for the waiting threads, the next time they evaluate it. > > > > > > > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > > > > releases c.lock, and tries to acquire m. The problem arises > > > > because the second waiter can get to these steps before the first > > > > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > > > > second to reach :308 will find c.tickets # 0, as did the first, > > > > and proceed. > > > > > > > > Each will eventually, do a M3-wake. If this were Posix condition > > > > variables, this would be wrong, because Posix says only one > > > > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > > > > says one or more waiters proceed. However, each will, in the > > > > process, get to :322, and DEC(c.tickets). Signal only provided > > > > one ticket, but the waiters have taken two. c.tickets goes > > > > negative. > > > > > > > > After this, when some thread can do another (M3-)Wait, and some > > > > other does another (M3-)Signal, c.tickets increments only back to > > > > zero, which means the new waiter will not M3-wake, even though it > > > > should. > > > > > > > > > > > > I am going to try to construct a test case that will force > > > > this scenario. > > > > > > > > See more below: > > > > > > > > > > > > On 07/18/2016 08:53 PM, Jay wrote: > > > > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > > > > > > > Otherwise previous version I think had a very large lock. > > > > > > > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > > > > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > > > > > > > I guess you are saying it would be unfair, which is ok. > > > > > > > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > > > > > > > I'll check on the unused event. > > > > > > > > > > - Jay > > > > > > > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > > > > >> comments: > > > > >> > > > > >> This looks like a very direct implementation of Schmidts' generation count > > > > >> algorithm, with a number of important little details additionally taken care > > > > >> of. > > > > >> > > > > >> On small difference is at line 308 (B, below) where Schmidt does > > > > >> c.counter>count instead of #. Usually, this would make no difference, > > > > >> since counter only increases, but on a long-running program, it could > > > > >> overflow. If this happened,as it often does, as silent wrap-around to > > > > >> FIRST(INTEGER), some threads could be trapped for an extremely long > > > > >> time, waiting for counter come back around greater than their count. As > > > > >> coded in ThreadWin32, they would be allowed to proceed normally. > > > > >> > > > > >> The only glitch would be that, if there were simultaneously waiting threads > > > > >> separated by NUMBER(INTEGER) truly different generations (but the same value > > > > >> of c.counter), the recent ones could unfairly proceed along with the > > > > >> extremely old ones. That seems extremely less likely than the > > > > >> mere existence of an overflow. > > > > >> > > > > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > > > > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > > > > >> only created at :537 and deleted at :554, but never used in any real > > > > >> way. It could be deleted. > > > > >> > > > > >> As for this code: > > > > >> > > > > >> EnterCriticalSection(conditionLock); > > > > >> > > > > >> (* Capture the value of the counter before we start waiting. > > > > >> * We will not stop waiting until the counter changes. > > > > >> * That is, we will not stop waiting until a signal > > > > >> * comes in after we start waiting. > > > > >> *) > > > > >> > > > > >> count := c.counter; > > > > >> INC(c.waiters); > > > > >> > > > > >> LeaveCriticalSection(conditionLock); > > > > >> m.release(); (* can this be moved to before taking conditionLock? *) > > > > >> > > > > >> No, it is important to sample and save c.counter in the order threads wait. > > > > >> Retaining m until this is done ensures this. Otherwise, some other thread > > > > >> that waited later could have altered c.counter before this one gets conditionLock > > > > >> and saves its copy of c.counter. > > > > >> > > > > >> I have made several comment changes to ThreadWin32.m3 that would have made > > > > >> it easier to vet. I will commit these, but only comments. It is either > > > > >> very difficult or impossible for me to test or even recompile this module, > > > > >> and, especially as fragile as this kind of code is, I don't want to commit > > > > >> any substantive changes untested and uncompiled. > > > > >> > > > > >> More below: > > > > >> > > > > >>> On 07/07/2016 04:24 AM, Jay K wrote: > > > > >>> So...I do NOT understand all of this. > > > > >>> > > > > >>> > > > > >>> However, comparing the three implementations > > > > >>> and attempting to understand ours, > > > > >>> I am struck by the following > > > > >>> > > > > >>> - Our broadcast seems ok, but > > > > >>> - our signal seems to wake everyone, > > > > >>> and then...they race a bit, > > > > >>> one will decide it is last, and > > > > >>> reset the event, but every prior waiter > > > > >>> is still woken. > > > > >>> Why not merge the following chunks: > > > > >> > > > > >> I don't think this will work. It is important that, once a thread has decided, > > > > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > > > > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > > > > >> if it is also the last of its generation to proceed, resets c.waitevent. This > > > > >> is ensured by its having already reacquired m first, preventing any other thread > > > > >> from returning from its Wait yet. > > > > > > > > I was wrong about this. There is already no fairness being enforced here. Even with > > > > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > > > > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > > > > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > > > > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > > > > > > > Merging, as you proposed, is probably close to the fix for this bug. > > > > > > > > > > > > >> > > > > >> And it can't attempt to acquire m while already holding conditionLock, because this would > > > > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > > > > >> already holding m, at line 265. > > > > >> > > > > >>> > > > > >>> > > > > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > > > > >>> . > > > > >>> . > > > > >>> . > > > > >>> 1 > > > > >>> EnterCriticalSection(conditionLock); > > > > >>> waitDone := (c.tickets # 0 AND c.counter # count); > > > > >> A:---------^ > > > > >> B:--------------------------------------------------^ > > > > >>> LeaveCriticalSection(conditionLock); > > > > >>> END; (* WHILE *) > > > > >>> IF waitDone THEN > > > > >>> alerted := FALSE; > > > > >>> END; > > > > >>> m.acquire(); > > > > >>> 2 > > > > >>> EnterCriticalSection(conditionLock); > > > > >>> DEC(c.waiters); > > > > >>> IF waitDone THEN esp. here. > > > > >>> DEC(c.tickets); > > > > >> > > > > >> C:---------------^ > > > > >>> lastWaiter := (c.tickets = 0); > > > > >>> END; > > > > >>> LeaveCriticalSection(conditionLock); > > > > >>> > > > > >>> > > > > >>> That is, if we decrement tickets earlier > > > > >>> within the first critical section, > > > > >>> while we will still wake everyone, > > > > >>> only one will decide waitDone and the rest will keep looping. > > > > >>> A downside of this, perhaps, is that waking all for Broadcast > > > > >>> might be a little slower. > > > > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > > > > >>> implementations, they seem to deal with this differently than us, > > > > >>> and they each do about the same thing -- they use a counted semaphore. > > > > >>> In the boost case, it appears they duplicate the semaphore for every > > > > >>> notification generation, which seems expensive. > > > > >>> > > > > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > > > > >>> Or they can do their own reference counting? > > > > >>> jdk7 seems to looke like jdk6. > > > > >>> The code is gone in jdk8 and I can't easily find the delete in history. > > > > >>> The lock merging jdk does probably helps here too. > > > > >>> In fact they merge #1 and #2 above as a result. > > > > >>> But they still initially wake all threads for signal, not just broadcast. > > > > >>> There is also the problem that all the event waits > > > > >>> are followed by EnterCriticalSection (or jdk facsimile). > > > > >>> - Jay > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > -----! > > > > --! > > > > >> --- > > > > >>> From: jay.krell at cornell.edu > > > > >>> To: m3devel at elegosoft.com > > > > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > > > > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > > > > >>> > > > > >>> Here is another implementation to consider: > > > > >>> > > > > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > > > > >>> > > > > >>> - Jay > > > > >>> > > > > >>> > > > > >>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > -----! > > > > --! > > > > >> --- > > > > >>> From: jay.krell at cornell.edu > > > > >>> To: m3devel at elegosoft.com > > > > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > > > > >>> Subject: [M3devel] purported condition variable problems on Win32? > > > > >>> > > > > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > > > > >>> > > > > >>> vs. > > > > >>> > > > > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > > > >>> vs. > > > > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > > > > >>> I wrote this: > > > > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > > > > >>> alertable: BOOLEAN) RAISES {Alerted} = > > > > >>> (* LL = m on entry and exit, but not for the duration > > > > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > > > > >>> * NOTE that they merge the user lock and the condition lock. > > > > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > > > >>> * "3.3. The Generation Count Solution" > > > > >>> *) > > > > >>> > > > > >>> Do we have the problems described in README.CV? > > > > >>> > > > > >>> I haven't looked through the ACE code to see > > > > >>> to what extent they resemble solution 3.3, or if they > > > > >>> changed as a result of this discussion -- which I admit I don't understand > > > > >>> and haven't read closely. > > > > >>> > > > > >>> > > > > >>> Spurious wakeups are ok, though should be minimized. > > > > >>> > > > > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > > > >>> > > > > >>> Thank you, > > > > >>> - Jay > > > > >>> > > > > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > > > >>> > > > > >>> > > > > >>> _______________________________________________ > > > > >>> M3devel mailing list > > > > >>> M3devel at elegosoft.com > > > > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > > > >> > > > > >> -- > > > > >> Rodney Bates > > > > >> rodney.m.bates at acm.org > > > > >> > > > > >> > > > > > > > > > > > > > -- > > > > Rodney Bates > > > > rodney.m.bates at acm.org > > > > -- > > Rodney Bates > > rodney.m.bates at acm.org -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Sun Jul 24 23:53:20 2016 From: jay.krell at cornell.edu (Jay) Date: Sun, 24 Jul 2016 21:53:20 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: <5794ECAE.5070308@lcwb.coop> References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop> <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> <578FAA2F.6020105@lcwb.coop> <5793DB37.10603@lcwb.coop> <5794ECAE.5070308@lcwb.coop> Message-ID: It is a problem for merging mutex with condition lock, or rather, for having Wait rely on the external lock. - Jay > On Jul 24, 2016, at 9:28 AM, "Rodney M. Bates" wrote: > > No, they don't. But I don't think that presents a problem. > >> On 07/23/2016 06:37 PM, Jay K wrote: >> I don't think Signal/Broadcast require caller be holding any lock, is a problem. >> >> - Jay >> >> >> >> > Date: Sat, 23 Jul 2016 16:01:43 -0500 >> > From: rodney_bates at lcwb.coop >> > To: jay.krell at cornell.edu; rodney.m.bates at acm.org >> > CC: m3devel at elegosoft.com >> > Subject: Re: [M3devel] purported condition variable problems on Win32? >> > >> > >> > >> > On 07/22/2016 03:20 AM, Jay K wrote: >> > > Here is very weak circumstantial evidence that the locks can be merged: >> > > >> > > >> > > >> > > https://msdn.microsoft.com/en-us/library/windows/desktop/ms682052(v=vs.85).aspx >> > > >> > > "It is often convenient to use more than one condition variable with the same lock. >> > > For example, an implementation of a reader/writer lock might use a single critical >> > > section but separate condition variables for readers and writers. >> > > " >> > > >> > > >> > > i.e. the mapping isn't one lock to one condition, but the convenient >> > > construct is multiple conditions to one lock, not multiple locks to one condition. >> > > >> > > >> > > http://linux.die.net/man/3/pthread_cond_wait >> > > >> > > " >> > > The effect of using more than one mutex for concurrent pthread_cond_timedwait() or >> > > pthread_cond_wait() operations on the same condition variable is undefined; that is, >> > > a condition variable becomes bound to a unique mutex when a thread waits on the condition >> > > variable, and this (dynamic) binding shall end when the wait returns. >> > > " >> > > >> > >> > Yes, these are the usual rules about the relationship between mutexs and condition variables. >> > There is sometimes a definite need for >1 condition variable associated with a single >> > mutex. The other way around might be definable, but I think would create a big mess. >> > >> > Our Thread interface documents only that Wait(m,c) must have m locked at the time of >> > call (and will return with it locked too, but not necessarily for the duration. >> > >> > Chapter 4 of SPwM3, "An Introduction to Programming with Threads" informally defines >> > a _monitor_ as "some data, a mutex, and zero or more condition variables" and says >> > further "A particular condition variable is always used in conjunction with the same >> > mutex and its data". (SPwM3, 4.3.2, p93) >> > >> > It looks to me like the formal specification of Wait, Signal, and Broadcast in >> > SRwM3, 5.3.3, p124 would allow a condition to simultaneously contain waiters >> > that passed in different mutexes. Each thread remembers locally what mutex it >> > wants to reacquire when it wakes up. >> > >> > I'm not sure we really want to commit to that. Client code's doing it would no doubt >> > create even trickier code than this kind of stuff already is. Off hand, I have a >> > hard time thinking of a reasonable use-case. >> > >> > If we decide to go with the informal statement, the implementations really should >> > enforce it by storing the associated mutex in the condition variable, and verifying >> > that additional waiters supply the same one. >> > >> > Probably, in most implementations, you could get away with moving the association >> > of a condition variable from one mutex to another when no threads were waiting on >> > the condition. Buy\t the only possible benefit I can think of would be memory savings, >> > not needing as many condition variables, since the different mutex associations would >> > be, well, mutually exclusive. >> > >> > >> > >> > >> > > i.e. while the lock used with a condition variable can change through time, >> > > there can be only one at a time. >> > > >> > > And then, presumably win32 condition variables and posix condition variables >> > > and Modula-3 Thread.Condition are all kind of similar. >> > > >> > > Hm, ok, a problem with merging the locks, is it leaves no lock for Signal/Broadcast. >> > > Perhaps they should use atomic/Interlocked. >> > > >> > > - Jay >> > > >> > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > -----! >> > --- >> > > From: jay.krell at cornell.edu >> > > To: rodney.m.bates at acm.org >> > > CC: m3devel at elegosoft.com >> > > Subject: RE: [M3devel] purported condition variable problems on Win32? >> > > Date: Thu, 21 Jul 2016 06:43:16 +0000 >> > > >> > > That question was meant for the other bug. >> > > >> > > Can this one be fixed by rechecking if tickets is positive before decrementing it? >> > > This seems like a big problem in Schmidt's code. >> > > >> > > I'm nervous about merging the locks, but maybe. >> > > In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. >> > > Though I realize that is actually necessarily part of what gets done. >> > > Schmidt clearly did not merge the locks, and the Java version does. >> > > The Java version also has its own simple and not-terrible critical section. >> > > The Java version also disappeared in newer releases and I didn't track down what happened to it. >> > > >> > > - Jay >> > > >> > > >> > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > -----! >> > --- >> > > From: jay.krell at cornell.edu >> > > To: rodney.m.bates at acm.org >> > > CC: m3devel at elegosoft.com >> > > Subject: RE: [M3devel] purported condition variable problems on Win32? >> > > Date: Thu, 21 Jul 2016 06:35:41 +0000 >> > > >> > > Am I understanding this? >> > > There are some waiters. >> > > There is a broadcast ("wake all"). Some waiters are woken. >> > > There are more waiters. >> > > There is a signal ("wake one"). >> > > There is a chance that some of the original waiters will remain waiting while the later waiters are not. >> > > That is, the signaled threads kind of steal the wake intended by the broadcast. >> > > >> > > >> > > Is this unfairness or incorrectness? >> > > It feels like unfairness. >> > > It feels like condition variables are specified fairly weakly. >> > > >> > > >> > > - Jay >> > > >> > > >> > > > Date: Wed, 20 Jul 2016 11:43:27 -0500 >> > > > From: rodney_bates at lcwb.coop >> > > > To: jay.krell at cornell.edu; rodney.m.bates at acm.org >> > > > CC: m3devel at elegosoft.com >> > > > Subject: Re: [M3devel] purported condition variable problems on Win32? >> > > > >> > > > In looking at ThreadWin32.m3 more, I think I see a significant bug. >> > > > >> > > > This gets confusing, since there are Modula-3 waits, signals, >> > > > etc. and similar concepts in Win32, but which are not the >> > > > same. I am going the try to put "M3-" and "Win-" prefixes on >> > > > things to make this clearer. I cartainly need to do this >> > > > for my own benefit, if for nobody else. >> > > > >> > > > Suppose two threads are both M3-waiting in Wait(m,c,...), further >> > > > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. >> > > > c.tickets = 0, and c.waiters = 2. >> > > > >> > > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), >> > > > which will allow both the waiters to Win-wakeup sometime soon. >> > > > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly >> > > > to eventually allow only one waiter to M3-wakeup) and increments >> > > > c.counter, setting things up so that the waitDone expression at :308 >> > > > will be TRUE for the waiting threads, the next time they evaluate it. >> > > > >> > > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, >> > > > releases c.lock, and tries to acquire m. The problem arises >> > > > because the second waiter can get to these steps before the first >> > > > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The >> > > > second to reach :308 will find c.tickets # 0, as did the first, >> > > > and proceed. >> > > > >> > > > Each will eventually, do a M3-wake. If this were Posix condition >> > > > variables, this would be wrong, because Posix says only one >> > > > waiter proceeds. For M3, it's OK, if unnecessary, because M3 >> > > > says one or more waiters proceed. However, each will, in the >> > > > process, get to :322, and DEC(c.tickets). Signal only provided >> > > > one ticket, but the waiters have taken two. c.tickets goes >> > > > negative. >> > > > >> > > > After this, when some thread can do another (M3-)Wait, and some >> > > > other does another (M3-)Signal, c.tickets increments only back to >> > > > zero, which means the new waiter will not M3-wake, even though it >> > > > should. >> > > > >> > > > >> > > > I am going to try to construct a test case that will force >> > > > this scenario. >> > > > >> > > > See more below: >> > > > >> > > > >> > > > On 07/18/2016 08:53 PM, Jay wrote: >> > > > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. >> > > > > >> > > > > Otherwise previous version I think had a very large lock. >> > > > > >> > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. >> > > > > >> > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. >> > > > > >> > > > > I guess you are saying it would be unfair, which is ok. >> > > > > >> > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. >> > > > > >> > > > > I'll check on the unused event. >> > > > > >> > > > > - Jay >> > > > > >> > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> I spent some time looking over the signal part of ThreadWin32.m3. A few >> > > > >> comments: >> > > > >> >> > > > >> This looks like a very direct implementation of Schmidts' generation count >> > > > >> algorithm, with a number of important little details additionally taken care >> > > > >> of. >> > > > >> >> > > > >> On small difference is at line 308 (B, below) where Schmidt does >> > > > >> c.counter>count instead of #. Usually, this would make no difference, >> > > > >> since counter only increases, but on a long-running program, it could >> > > > >> overflow. If this happened,as it often does, as silent wrap-around to >> > > > >> FIRST(INTEGER), some threads could be trapped for an extremely long >> > > > >> time, waiting for counter come back around greater than their count. As >> > > > >> coded in ThreadWin32, they would be allowed to proceed normally. >> > > > >> >> > > > >> The only glitch would be that, if there were simultaneously waiting threads >> > > > >> separated by NUMBER(INTEGER) truly different generations (but the same value >> > > > >> of c.counter), the recent ones could unfairly proceed along with the >> > > > >> extremely old ones. That seems extremely less likely than the >> > > > >> mere existence of an overflow. >> > > > >> >> > > > >> Both type Condition and type Activation have a field waitEvent: HANDLE. >> > > > >> Condition.waitEvent is extensively used, but Activation.waitEvent is >> > > > >> only created at :537 and deleted at :554, but never used in any real >> > > > >> way. It could be deleted. >> > > > >> >> > > > >> As for this code: >> > > > >> >> > > > >> EnterCriticalSection(conditionLock); >> > > > >> >> > > > >> (* Capture the value of the counter before we start waiting. >> > > > >> * We will not stop waiting until the counter changes. >> > > > >> * That is, we will not stop waiting until a signal >> > > > >> * comes in after we start waiting. >> > > > >> *) >> > > > >> >> > > > >> count := c.counter; >> > > > >> INC(c.waiters); >> > > > >> >> > > > >> LeaveCriticalSection(conditionLock); >> > > > >> m.release(); (* can this be moved to before taking conditionLock? *) >> > > > >> >> > > > >> No, it is important to sample and save c.counter in the order threads wait. >> > > > >> Retaining m until this is done ensures this. Otherwise, some other thread >> > > > >> that waited later could have altered c.counter before this one gets conditionLock >> > > > >> and saves its copy of c.counter. >> > > > >> >> > > > >> I have made several comment changes to ThreadWin32.m3 that would have made >> > > > >> it easier to vet. I will commit these, but only comments. It is either >> > > > >> very difficult or impossible for me to test or even recompile this module, >> > > > >> and, especially as fragile as this kind of code is, I don't want to commit >> > > > >> any substantive changes untested and uncompiled. >> > > > >> >> > > > >> More below: >> > > > >> >> > > > >>> On 07/07/2016 04:24 AM, Jay K wrote: >> > > > >>> So...I do NOT understand all of this. >> > > > >>> >> > > > >>> >> > > > >>> However, comparing the three implementations >> > > > >>> and attempting to understand ours, >> > > > >>> I am struck by the following >> > > > >>> >> > > > >>> - Our broadcast seems ok, but >> > > > >>> - our signal seems to wake everyone, >> > > > >>> and then...they race a bit, >> > > > >>> one will decide it is last, and >> > > > >>> reset the event, but every prior waiter >> > > > >>> is still woken. >> > > > >>> Why not merge the following chunks: >> > > > >> >> > > > >> I don't think this will work. It is important that, once a thread has decided, >> > > > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed >> > > > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, >> > > > >> if it is also the last of its generation to proceed, resets c.waitevent. This >> > > > >> is ensured by its having already reacquired m first, preventing any other thread >> > > > >> from returning from its Wait yet. >> > > > >> > > > I was wrong about this. There is already no fairness being enforced here. Even with >> > > > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is >> > > > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. >> > > > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters >> > > > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. >> > > > >> > > > Merging, as you proposed, is probably close to the fix for this bug. >> > > > >> > > > >> > > > >> >> > > > >> And it can't attempt to acquire m while already holding conditionLock, because this would >> > > > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while >> > > > >> already holding m, at line 265. >> > > > >> >> > > > >>> >> > > > >>> >> > > > >>> WHILE (NOT alerted) AND (NOT waitDone) DO >> > > > >>> . >> > > > >>> . >> > > > >>> . >> > > > >>> 1 >> > > > >>> EnterCriticalSection(conditionLock); >> > > > >>> waitDone := (c.tickets # 0 AND c.counter # count); >> > > > >> A:---------^ >> > > > >> B:--------------------------------------------------^ >> > > > >>> LeaveCriticalSection(conditionLock); >> > > > >>> END; (* WHILE *) >> > > > >>> IF waitDone THEN >> > > > >>> alerted := FALSE; >> > > > >>> END; >> > > > >>> m.acquire(); >> > > > >>> 2 >> > > > >>> EnterCriticalSection(conditionLock); >> > > > >>> DEC(c.waiters); >> > > > >>> IF waitDone THEN esp. here. >> > > > >>> DEC(c.tickets); >> > > > >> >> > > > >> C:---------------^ >> > > > >>> lastWaiter := (c.tickets = 0); >> > > > >>> END; >> > > > >>> LeaveCriticalSection(conditionLock); >> > > > >>> >> > > > >>> >> > > > >>> That is, if we decrement tickets earlier >> > > > >>> within the first critical section, >> > > > >>> while we will still wake everyone, >> > > > >>> only one will decide waitDone and the rest will keep looping. >> > > > >>> A downside of this, perhaps, is that waking all for Broadcast >> > > > >>> might be a little slower. >> > > > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads >> > > > >>> implementations, they seem to deal with this differently than us, >> > > > >>> and they each do about the same thing -- they use a counted semaphore. >> > > > >>> In the boost case, it appears they duplicate the semaphore for every >> > > > >>> notification generation, which seems expensive. >> > > > >>> >> > > > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. >> > > > >>> Or they can do their own reference counting? >> > > > >>> jdk7 seems to looke like jdk6. >> > > > >>> The code is gone in jdk8 and I can't easily find the delete in history. >> > > > >>> The lock merging jdk does probably helps here too. >> > > > >>> In fact they merge #1 and #2 above as a result. >> > > > >>> But they still initially wake all threads for signal, not just broadcast. >> > > > >>> There is also the problem that all the event waits >> > > > >>> are followed by EnterCriticalSection (or jdk facsimile). >> > > > >>> - Jay >> > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > ----! >> > -----! >> > > > --! >> > > > >> --- >> > > > >>> From: jay.krell at cornell.edu >> > > > >>> To: m3devel at elegosoft.com >> > > > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? >> > > > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 >> > > > >>> >> > > > >>> Here is another implementation to consider: >> > > > >>> >> > > > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp >> > > > >>> >> > > > >>> - Jay >> > > > >>> >> > > > >>> >> > > > >>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > ----! >> > -----! >> > > > --! >> > > > >> --- >> > > > >>> From: jay.krell at cornell.edu >> > > > >>> To: m3devel at elegosoft.com >> > > > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 >> > > > >>> Subject: [M3devel] purported condition variable problems on Win32? >> > > > >>> >> > > > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV >> > > > >>> >> > > > >>> vs. >> > > > >>> >> > > > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >> > > > >>> vs. >> > > > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain >> > > > >>> I wrote this: >> > > > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; >> > > > >>> alertable: BOOLEAN) RAISES {Alerted} = >> > > > >>> (* LL = m on entry and exit, but not for the duration >> > > > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp >> > > > >>> * NOTE that they merge the user lock and the condition lock. >> > > > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >> > > > >>> * "3.3. The Generation Count Solution" >> > > > >>> *) >> > > > >>> >> > > > >>> Do we have the problems described in README.CV? >> > > > >>> >> > > > >>> I haven't looked through the ACE code to see >> > > > >>> to what extent they resemble solution 3.3, or if they >> > > > >>> changed as a result of this discussion -- which I admit I don't understand >> > > > >>> and haven't read closely. >> > > > >>> >> > > > >>> >> > > > >>> Spurious wakeups are ok, though should be minimized. >> > > > >>> >> > > > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. >> > > > >>> >> > > > >>> Thank you, >> > > > >>> - Jay >> > > > >>> >> > > > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> > > > >>> >> > > > >>> >> > > > >>> _______________________________________________ >> > > > >>> M3devel mailing list >> > > > >>> M3devel at elegosoft.com >> > > > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> > > > >> >> > > > >> -- >> > > > >> Rodney Bates >> > > > >> rodney.m.bates at acm.org >> > > > >> >> > > > >> >> > > > > >> > > > >> > > > -- >> > > > Rodney Bates >> > > > rodney.m.bates at acm.org >> > >> > -- >> > Rodney Bates >> > rodney.m.bates at acm.org > > -- > Rodney Bates > rodney.m.bates at acm.org From jay.krell at cornell.edu Mon Jul 25 07:50:56 2016 From: jay.krell at cornell.edu (Jay K) Date: Mon, 25 Jul 2016 05:50:56 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> <578FAA2F.6020105@lcwb.coop>, , , <5793DB37.10603@lcwb.coop>, <5794ECAE.5070308@lcwb.coop>, Message-ID: btw, I think you are right. In particular, regarding the "directed notify" pattern,and the fact that we can have a per-thread event. I think you don't see this excact implementation stategy...you see people creating an event per wait...is becausewe have the feature and defect that we require ourown runtime to create our threads, and we don't interoperatewell with threads created otherwise...and there is a fix for this though,at least on Windows (where we are discussing anyway) viam3core.dll's DllMain, which we don't implement but probably should. So, let's go ahead and put support in Thread.T. or Thread.Activationand solve this? - Jay > From: jay.krell at cornell.edu > CC: jay.krell at cornell.edu; m3devel at elegosoft.com > Subject: Re: [M3devel] purported condition variable problems on Win32? > Date: Sun, 24 Jul 2016 14:53:12 -0700 > To: rodney.m.bates at acm.org > > It is a problem for merging mutex with condition lock, or rather, for having Wait rely on the external lock. > > - Jay > > > On Jul 24, 2016, at 9:28 AM, "Rodney M. Bates" wrote: > > > > No, they don't. But I don't think that presents a problem. > > > >> On 07/23/2016 06:37 PM, Jay K wrote: > >> I don't think Signal/Broadcast require caller be holding any lock, is a problem. > >> > >> - Jay > >> > >> > >> > >> > Date: Sat, 23 Jul 2016 16:01:43 -0500 > >> > From: rodney_bates at lcwb.coop > >> > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > >> > CC: m3devel at elegosoft.com > >> > Subject: Re: [M3devel] purported condition variable problems on Win32? > >> > > >> > > >> > > >> > On 07/22/2016 03:20 AM, Jay K wrote: > >> > > Here is very weak circumstantial evidence that the locks can be merged: > >> > > > >> > > > >> > > > >> > > https://msdn.microsoft.com/en-us/library/windows/desktop/ms682052(v=vs.85).aspx > >> > > > >> > > "It is often convenient to use more than one condition variable with the same lock. > >> > > For example, an implementation of a reader/writer lock might use a single critical > >> > > section but separate condition variables for readers and writers. > >> > > " > >> > > > >> > > > >> > > i.e. the mapping isn't one lock to one condition, but the convenient > >> > > construct is multiple conditions to one lock, not multiple locks to one condition. > >> > > > >> > > > >> > > http://linux.die.net/man/3/pthread_cond_wait > >> > > > >> > > " > >> > > The effect of using more than one mutex for concurrent pthread_cond_timedwait() or > >> > > pthread_cond_wait() operations on the same condition variable is undefined; that is, > >> > > a condition variable becomes bound to a unique mutex when a thread waits on the condition > >> > > variable, and this (dynamic) binding shall end when the wait returns. > >> > > " > >> > > > >> > > >> > Yes, these are the usual rules about the relationship between mutexs and condition variables. > >> > There is sometimes a definite need for >1 condition variable associated with a single > >> > mutex. The other way around might be definable, but I think would create a big mess. > >> > > >> > Our Thread interface documents only that Wait(m,c) must have m locked at the time of > >> > call (and will return with it locked too, but not necessarily for the duration. > >> > > >> > Chapter 4 of SPwM3, "An Introduction to Programming with Threads" informally defines > >> > a _monitor_ as "some data, a mutex, and zero or more condition variables" and says > >> > further "A particular condition variable is always used in conjunction with the same > >> > mutex and its data". (SPwM3, 4.3.2, p93) > >> > > >> > It looks to me like the formal specification of Wait, Signal, and Broadcast in > >> > SRwM3, 5.3.3, p124 would allow a condition to simultaneously contain waiters > >> > that passed in different mutexes. Each thread remembers locally what mutex it > >> > wants to reacquire when it wakes up. > >> > > >> > I'm not sure we really want to commit to that. Client code's doing it would no doubt > >> > create even trickier code than this kind of stuff already is. Off hand, I have a > >> > hard time thinking of a reasonable use-case. > >> > > >> > If we decide to go with the informal statement, the implementations really should > >> > enforce it by storing the associated mutex in the condition variable, and verifying > >> > that additional waiters supply the same one. > >> > > >> > Probably, in most implementations, you could get away with moving the association > >> > of a condition variable from one mutex to another when no threads were waiting on > >> > the condition. Buy\t the only possible benefit I can think of would be memory savings, > >> > not needing as many condition variables, since the different mutex associations would > >> > be, well, mutually exclusive. > >> > > >> > > >> > > >> > > >> > > i.e. while the lock used with a condition variable can change through time, > >> > > there can be only one at a time. > >> > > > >> > > And then, presumably win32 condition variables and posix condition variables > >> > > and Modula-3 Thread.Condition are all kind of similar. > >> > > > >> > > Hm, ok, a problem with merging the locks, is it leaves no lock for Signal/Broadcast. > >> > > Perhaps they should use atomic/Interlocked. > >> > > > >> > > - Jay > >> > > > >> > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > -----! > >> > --- > >> > > From: jay.krell at cornell.edu > >> > > To: rodney.m.bates at acm.org > >> > > CC: m3devel at elegosoft.com > >> > > Subject: RE: [M3devel] purported condition variable problems on Win32? > >> > > Date: Thu, 21 Jul 2016 06:43:16 +0000 > >> > > > >> > > That question was meant for the other bug. > >> > > > >> > > Can this one be fixed by rechecking if tickets is positive before decrementing it? > >> > > This seems like a big problem in Schmidt's code. > >> > > > >> > > I'm nervous about merging the locks, but maybe. > >> > > In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. > >> > > Though I realize that is actually necessarily part of what gets done. > >> > > Schmidt clearly did not merge the locks, and the Java version does. > >> > > The Java version also has its own simple and not-terrible critical section. > >> > > The Java version also disappeared in newer releases and I didn't track down what happened to it. > >> > > > >> > > - Jay > >> > > > >> > > > >> > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > -----! > >> > --- > >> > > From: jay.krell at cornell.edu > >> > > To: rodney.m.bates at acm.org > >> > > CC: m3devel at elegosoft.com > >> > > Subject: RE: [M3devel] purported condition variable problems on Win32? > >> > > Date: Thu, 21 Jul 2016 06:35:41 +0000 > >> > > > >> > > Am I understanding this? > >> > > There are some waiters. > >> > > There is a broadcast ("wake all"). Some waiters are woken. > >> > > There are more waiters. > >> > > There is a signal ("wake one"). > >> > > There is a chance that some of the original waiters will remain waiting while the later waiters are not. > >> > > That is, the signaled threads kind of steal the wake intended by the broadcast. > >> > > > >> > > > >> > > Is this unfairness or incorrectness? > >> > > It feels like unfairness. > >> > > It feels like condition variables are specified fairly weakly. > >> > > > >> > > > >> > > - Jay > >> > > > >> > > > >> > > > Date: Wed, 20 Jul 2016 11:43:27 -0500 > >> > > > From: rodney_bates at lcwb.coop > >> > > > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > >> > > > CC: m3devel at elegosoft.com > >> > > > Subject: Re: [M3devel] purported condition variable problems on Win32? > >> > > > > >> > > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > >> > > > > >> > > > This gets confusing, since there are Modula-3 waits, signals, > >> > > > etc. and similar concepts in Win32, but which are not the > >> > > > same. I am going the try to put "M3-" and "Win-" prefixes on > >> > > > things to make this clearer. I cartainly need to do this > >> > > > for my own benefit, if for nobody else. > >> > > > > >> > > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > >> > > > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > >> > > > c.tickets = 0, and c.waiters = 2. > >> > > > > >> > > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > >> > > > which will allow both the waiters to Win-wakeup sometime soon. > >> > > > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > >> > > > to eventually allow only one waiter to M3-wakeup) and increments > >> > > > c.counter, setting things up so that the waitDone expression at :308 > >> > > > will be TRUE for the waiting threads, the next time they evaluate it. > >> > > > > >> > > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > >> > > > releases c.lock, and tries to acquire m. The problem arises > >> > > > because the second waiter can get to these steps before the first > >> > > > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > >> > > > second to reach :308 will find c.tickets # 0, as did the first, > >> > > > and proceed. > >> > > > > >> > > > Each will eventually, do a M3-wake. If this were Posix condition > >> > > > variables, this would be wrong, because Posix says only one > >> > > > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > >> > > > says one or more waiters proceed. However, each will, in the > >> > > > process, get to :322, and DEC(c.tickets). Signal only provided > >> > > > one ticket, but the waiters have taken two. c.tickets goes > >> > > > negative. > >> > > > > >> > > > After this, when some thread can do another (M3-)Wait, and some > >> > > > other does another (M3-)Signal, c.tickets increments only back to > >> > > > zero, which means the new waiter will not M3-wake, even though it > >> > > > should. > >> > > > > >> > > > > >> > > > I am going to try to construct a test case that will force > >> > > > this scenario. > >> > > > > >> > > > See more below: > >> > > > > >> > > > > >> > > > On 07/18/2016 08:53 PM, Jay wrote: > >> > > > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > >> > > > > > >> > > > > Otherwise previous version I think had a very large lock. > >> > > > > > >> > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > >> > > > > > >> > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > >> > > > > > >> > > > > I guess you are saying it would be unfair, which is ok. > >> > > > > > >> > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > >> > > > > > >> > > > > I'll check on the unused event. > >> > > > > > >> > > > > - Jay > >> > > > > > >> > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > >> > > > >> comments: > >> > > > >> > >> > > > >> This looks like a very direct implementation of Schmidts' generation count > >> > > > >> algorithm, with a number of important little details additionally taken care > >> > > > >> of. > >> > > > >> > >> > > > >> On small difference is at line 308 (B, below) where Schmidt does > >> > > > >> c.counter>count instead of #. Usually, this would make no difference, > >> > > > >> since counter only increases, but on a long-running program, it could > >> > > > >> overflow. If this happened,as it often does, as silent wrap-around to > >> > > > >> FIRST(INTEGER), some threads could be trapped for an extremely long > >> > > > >> time, waiting for counter come back around greater than their count. As > >> > > > >> coded in ThreadWin32, they would be allowed to proceed normally. > >> > > > >> > >> > > > >> The only glitch would be that, if there were simultaneously waiting threads > >> > > > >> separated by NUMBER(INTEGER) truly different generations (but the same value > >> > > > >> of c.counter), the recent ones could unfairly proceed along with the > >> > > > >> extremely old ones. That seems extremely less likely than the > >> > > > >> mere existence of an overflow. > >> > > > >> > >> > > > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > >> > > > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > >> > > > >> only created at :537 and deleted at :554, but never used in any real > >> > > > >> way. It could be deleted. > >> > > > >> > >> > > > >> As for this code: > >> > > > >> > >> > > > >> EnterCriticalSection(conditionLock); > >> > > > >> > >> > > > >> (* Capture the value of the counter before we start waiting. > >> > > > >> * We will not stop waiting until the counter changes. > >> > > > >> * That is, we will not stop waiting until a signal > >> > > > >> * comes in after we start waiting. > >> > > > >> *) > >> > > > >> > >> > > > >> count := c.counter; > >> > > > >> INC(c.waiters); > >> > > > >> > >> > > > >> LeaveCriticalSection(conditionLock); > >> > > > >> m.release(); (* can this be moved to before taking conditionLock? *) > >> > > > >> > >> > > > >> No, it is important to sample and save c.counter in the order threads wait. > >> > > > >> Retaining m until this is done ensures this. Otherwise, some other thread > >> > > > >> that waited later could have altered c.counter before this one gets conditionLock > >> > > > >> and saves its copy of c.counter. > >> > > > >> > >> > > > >> I have made several comment changes to ThreadWin32.m3 that would have made > >> > > > >> it easier to vet. I will commit these, but only comments. It is either > >> > > > >> very difficult or impossible for me to test or even recompile this module, > >> > > > >> and, especially as fragile as this kind of code is, I don't want to commit > >> > > > >> any substantive changes untested and uncompiled. > >> > > > >> > >> > > > >> More below: > >> > > > >> > >> > > > >>> On 07/07/2016 04:24 AM, Jay K wrote: > >> > > > >>> So...I do NOT understand all of this. > >> > > > >>> > >> > > > >>> > >> > > > >>> However, comparing the three implementations > >> > > > >>> and attempting to understand ours, > >> > > > >>> I am struck by the following > >> > > > >>> > >> > > > >>> - Our broadcast seems ok, but > >> > > > >>> - our signal seems to wake everyone, > >> > > > >>> and then...they race a bit, > >> > > > >>> one will decide it is last, and > >> > > > >>> reset the event, but every prior waiter > >> > > > >>> is still woken. > >> > > > >>> Why not merge the following chunks: > >> > > > >> > >> > > > >> I don't think this will work. It is important that, once a thread has decided, > >> > > > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > >> > > > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > >> > > > >> if it is also the last of its generation to proceed, resets c.waitevent. This > >> > > > >> is ensured by its having already reacquired m first, preventing any other thread > >> > > > >> from returning from its Wait yet. > >> > > > > >> > > > I was wrong about this. There is already no fairness being enforced here. Even with > >> > > > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > >> > > > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > >> > > > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > >> > > > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > >> > > > > >> > > > Merging, as you proposed, is probably close to the fix for this bug. > >> > > > > >> > > > > >> > > > >> > >> > > > >> And it can't attempt to acquire m while already holding conditionLock, because this would > >> > > > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > >> > > > >> already holding m, at line 265. > >> > > > >> > >> > > > >>> > >> > > > >>> > >> > > > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > >> > > > >>> . > >> > > > >>> . > >> > > > >>> . > >> > > > >>> 1 > >> > > > >>> EnterCriticalSection(conditionLock); > >> > > > >>> waitDone := (c.tickets # 0 AND c.counter # count); > >> > > > >> A:---------^ > >> > > > >> B:--------------------------------------------------^ > >> > > > >>> LeaveCriticalSection(conditionLock); > >> > > > >>> END; (* WHILE *) > >> > > > >>> IF waitDone THEN > >> > > > >>> alerted := FALSE; > >> > > > >>> END; > >> > > > >>> m.acquire(); > >> > > > >>> 2 > >> > > > >>> EnterCriticalSection(conditionLock); > >> > > > >>> DEC(c.waiters); > >> > > > >>> IF waitDone THEN esp. here. > >> > > > >>> DEC(c.tickets); > >> > > > >> > >> > > > >> C:---------------^ > >> > > > >>> lastWaiter := (c.tickets = 0); > >> > > > >>> END; > >> > > > >>> LeaveCriticalSection(conditionLock); > >> > > > >>> > >> > > > >>> > >> > > > >>> That is, if we decrement tickets earlier > >> > > > >>> within the first critical section, > >> > > > >>> while we will still wake everyone, > >> > > > >>> only one will decide waitDone and the rest will keep looping. > >> > > > >>> A downside of this, perhaps, is that waking all for Broadcast > >> > > > >>> might be a little slower. > >> > > > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > >> > > > >>> implementations, they seem to deal with this differently than us, > >> > > > >>> and they each do about the same thing -- they use a counted semaphore. > >> > > > >>> In the boost case, it appears they duplicate the semaphore for every > >> > > > >>> notification generation, which seems expensive. > >> > > > >>> > >> > > > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > >> > > > >>> Or they can do their own reference counting? > >> > > > >>> jdk7 seems to looke like jdk6. > >> > > > >>> The code is gone in jdk8 and I can't easily find the delete in history. > >> > > > >>> The lock merging jdk does probably helps here too. > >> > > > >>> In fact they merge #1 and #2 above as a result. > >> > > > >>> But they still initially wake all threads for signal, not just broadcast. > >> > > > >>> There is also the problem that all the event waits > >> > > > >>> are followed by EnterCriticalSection (or jdk facsimile). > >> > > > >>> - Jay > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > ----! > >> > -----! > >> > > > --! > >> > > > >> --- > >> > > > >>> From: jay.krell at cornell.edu > >> > > > >>> To: m3devel at elegosoft.com > >> > > > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > >> > > > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > >> > > > >>> > >> > > > >>> Here is another implementation to consider: > >> > > > >>> > >> > > > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > >> > > > >>> > >> > > > >>> - Jay > >> > > > >>> > >> > > > >>> > >> > > > >>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > ----! > >> > -----! > >> > > > --! > >> > > > >> --- > >> > > > >>> From: jay.krell at cornell.edu > >> > > > >>> To: m3devel at elegosoft.com > >> > > > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > >> > > > >>> Subject: [M3devel] purported condition variable problems on Win32? > >> > > > >>> > >> > > > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > >> > > > >>> > >> > > > >>> vs. > >> > > > >>> > >> > > > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >> > > > >>> vs. > >> > > > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > >> > > > >>> I wrote this: > >> > > > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > >> > > > >>> alertable: BOOLEAN) RAISES {Alerted} = > >> > > > >>> (* LL = m on entry and exit, but not for the duration > >> > > > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > >> > > > >>> * NOTE that they merge the user lock and the condition lock. > >> > > > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >> > > > >>> * "3.3. The Generation Count Solution" > >> > > > >>> *) > >> > > > >>> > >> > > > >>> Do we have the problems described in README.CV? > >> > > > >>> > >> > > > >>> I haven't looked through the ACE code to see > >> > > > >>> to what extent they resemble solution 3.3, or if they > >> > > > >>> changed as a result of this discussion -- which I admit I don't understand > >> > > > >>> and haven't read closely. > >> > > > >>> > >> > > > >>> > >> > > > >>> Spurious wakeups are ok, though should be minimized. > >> > > > >>> > >> > > > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > >> > > > >>> > >> > > > >>> Thank you, > >> > > > >>> - Jay > >> > > > >>> > >> > > > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >> > > > >>> > >> > > > >>> > >> > > > >>> _______________________________________________ > >> > > > >>> M3devel mailing list > >> > > > >>> M3devel at elegosoft.com > >> > > > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >> > > > >> > >> > > > >> -- > >> > > > >> Rodney Bates > >> > > > >> rodney.m.bates at acm.org > >> > > > >> > >> > > > >> > >> > > > > > >> > > > > >> > > > -- > >> > > > Rodney Bates > >> > > > rodney.m.bates at acm.org > >> > > >> > -- > >> > Rodney Bates > >> > rodney.m.bates at acm.org > > > > -- > > Rodney Bates > > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Jul 25 09:27:49 2016 From: jay.krell at cornell.edu (Jay K) Date: Mon, 25 Jul 2016 07:27:49 -0000 Subject: [M3devel] Grisu vs. NT386 Message-ID: Fyi, Grisu reveals at least two problems in the NT386 backend.In particular, it uses LONGINT, and we don't have much code that does. The two problem are that NT386: - generates function calls for e.g. 64bit integer muliplication - doesn't like generating function calls within function calls, i.e. VAR a,b: LONGINT; Foo(a * b); It fails an assert -- when it goes to do the multiply, which is a function call, it asserts that no function call is in progress. For now I'll change Grisu: VAR a,b,t: LONGINT; t := a * b; Foo(t); but fixing NT386 shouldn't be difficult. Really, we should allow for: Foo(Bar(a * b) * b); .. the thing is, the frontend handles this -- it produces temporaries for function return results: t1 := Bar(a * b); Foo(t1 * a * b); but it doesn't do so for things like add/subtract/multiply/divide. Perhaps it should.Decent backends, including possibly NT386, will enregister the temporaries and generate good code anyway. Thoughts? The alternative is NT386 needs to maintain a stack where it doesn't previously -- a stack of pending function calls. 2) Grisu uses loophole between LONGREAL and LONGINT. This is reasonable, but perhaps not previously seen.NT386 requires loopholes that convert between integers and floats to be using REAL, at least for one direction.This seem merely historical and it should allow LONGREAL, or in particular, it should require no size change.Historically NT386 backend maintained an internal stack, of 32bit sized things -- variables and constants.I extended that to be variably sized, i.e. 32bits or 64bits, but this code wasn't updated. It should be trivial, maybe just relaxing the assertion. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Mon Jul 25 18:18:29 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 25 Jul 2016 16:18:29 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop> <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> <578FAA2F.6020105@lcwb.coop> <5793DB37.10603@lcwb.coop> <5794ECAE.5070308@lcwb.coop> Message-ID: <57963BB2.3000602@lcwb.coop> On 07/24/2016 04:53 PM, Jay wrote: > It is a problem for merging mutex with condition lock, or rather, for having Wait rely on the external lock. > I guess I don't understand what you mean by this. We don't want the condition variable and the mutex to be the same object, because it can be desirable to have more than one condition (in the informal sense) that different threads waiting on the same mutex-protected data are waiting for. The classic example is a FIFO message buffer with limited capacity. Some threads want to receive a message and wait for one to be available. Others want to send, and wait for the buffer to have space for it. So you want one Condition variable for each condition and signallers Signal the one they just made TRUE. You could do this with only one condition variable that gets a Broadcast when "something has changed", with all waiters waking up and rechecking for what they want, but it's overly awkward and entails a whole lot of unnecessary wake/sleep looping. "lock" is pretty ambiguous here. Could at least mean M3-Mutex, M3-Condition-variable, Win-critical-region, Win-event. > - Jay > >> On Jul 24, 2016, at 9:28 AM, "Rodney M. Bates" wrote: >> >> No, they don't. But I don't think that presents a problem. >> >>> On 07/23/2016 06:37 PM, Jay K wrote: >>> I don't think Signal/Broadcast require caller be holding any lock, is a problem. >>> >>> - Jay >>> >>> >>> >>>> Date: Sat, 23 Jul 2016 16:01:43 -0500 >>>> From: rodney_bates at lcwb.coop >>>> To: jay.krell at cornell.edu; rodney.m.bates at acm.org >>>> CC: m3devel at elegosoft.com >>>> Subject: Re: [M3devel] purported condition variable problems on Win32? >>>> >>>> >>>> >>>> On 07/22/2016 03:20 AM, Jay K wrote: >>>>> Here is very weak circumstantial evidence that the locks can be merged: >>>>> >>>>> >>>>> >>>>> https://msdn.microsoft.com/en-us/library/windows/desktop/ms682052(v=vs.85).aspx >>>>> >>>>> "It is often convenient to use more than one condition variable with the same lock. >>>>> For example, an implementation of a reader/writer lock might use a single critical >>>>> section but separate condition variables for readers and writers. >>>>> " >>>>> >>>>> >>>>> i.e. the mapping isn't one lock to one condition, but the convenient >>>>> construct is multiple conditions to one lock, not multiple locks to one condition. >>>>> >>>>> >>>>> http://linux.die.net/man/3/pthread_cond_wait >>>>> >>>>> " >>>>> The effect of using more than one mutex for concurrent pthread_cond_timedwait() or >>>>> pthread_cond_wait() operations on the same condition variable is undefined; that is, >>>>> a condition variable becomes bound to a unique mutex when a thread waits on the condition >>>>> variable, and this (dynamic) binding shall end when the wait returns. >>>>> " >>>>> >>>> >>>> Yes, these are the usual rules about the relationship between mutexs and condition variables. >>>> There is sometimes a definite need for >1 condition variable associated with a single >>>> mutex. The other way around might be definable, but I think would create a big mess. >>>> >>>> Our Thread interface documents only that Wait(m,c) must have m locked at the time of >>>> call (and will return with it locked too, but not necessarily for the duration. >>>> >>>> Chapter 4 of SPwM3, "An Introduction to Programming with Threads" informally defines >>>> a _monitor_ as "some data, a mutex, and zero or more condition variables" and says >>>> further "A particular condition variable is always used in conjunction with the same >>>> mutex and its data". (SPwM3, 4.3.2, p93) >>>> >>>> It looks to me like the formal specification of Wait, Signal, and Broadcast in >>>> SRwM3, 5.3.3, p124 would allow a condition to simultaneously contain waiters >>>> that passed in different mutexes. Each thread remembers locally what mutex it >>>> wants to reacquire when it wakes up. >>>> >>>> I'm not sure we really want to commit to that. Client code's doing it would no doubt >>>> create even trickier code than this kind of stuff already is. Off hand, I have a >>>> hard time thinking of a reasonable use-case. >>>> >>>> If we decide to go with the informal statement, the implementations really should >>>> enforce it by storing the associated mutex in the condition variable, and verifying >>>> that additional waiters supply the same one. >>>> >>>> Probably, in most implementations, you could get away with moving the association >>>> of a condition variable from one mutex to another when no threads were waiting on >>>> the condition. Buy\t the only possible benefit I can think of would be memory savings, >>>> not needing as many condition variables, since the different mutex associations would >>>> be, well, mutually exclusive. >>>> >>>> >>>> >>>> >>>>> i.e. while the lock used with a condition variable can change through time, >>>>> there can be only one at a time. >>>>> >>>>> And then, presumably win32 condition variables and posix condition variables >>>>> and Modula-3 Thread.Condition are all kind of similar. >>>>> >>>>> Hm, ok, a problem with merging the locks, is it leaves no lock for Signal/Broadcast. >>>>> Perhaps they should use atomic/Interlocked. >>>>> >>>>> - Jay >>>>> >>>>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> -----! >>>> --- >>>>> From: jay.krell at cornell.edu >>>>> To: rodney.m.bates at acm.org >>>>> CC: m3devel at elegosoft.com >>>>> Subject: RE: [M3devel] purported condition variable problems on Win32? >>>>> Date: Thu, 21 Jul 2016 06:43:16 +0000 >>>>> >>>>> That question was meant for the other bug. >>>>> >>>>> Can this one be fixed by rechecking if tickets is positive before decrementing it? >>>>> This seems like a big problem in Schmidt's code. >>>>> >>>>> I'm nervous about merging the locks, but maybe. >>>>> In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. >>>>> Though I realize that is actually necessarily part of what gets done. >>>>> Schmidt clearly did not merge the locks, and the Java version does. >>>>> The Java version also has its own simple and not-terrible critical section. >>>>> The Java version also disappeared in newer releases and I didn't track down what happened to it. >>>>> >>>>> - Jay >>>>> >>>>> >>>>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> -----! >>>> --- >>>>> From: jay.krell at cornell.edu >>>>> To: rodney.m.bates at acm.org >>>>> CC: m3devel at elegosoft.com >>>>> Subject: RE: [M3devel] purported condition variable problems on Win32? >>>>> Date: Thu, 21 Jul 2016 06:35:41 +0000 >>>>> >>>>> Am I understanding this? >>>>> There are some waiters. >>>>> There is a broadcast ("wake all"). Some waiters are woken. >>>>> There are more waiters. >>>>> There is a signal ("wake one"). >>>>> There is a chance that some of the original waiters will remain waiting while the later waiters are not. >>>>> That is, the signaled threads kind of steal the wake intended by the broadcast. >>>>> >>>>> >>>>> Is this unfairness or incorrectness? >>>>> It feels like unfairness. >>>>> It feels like condition variables are specified fairly weakly. >>>>> >>>>> >>>>> - Jay >>>>> >>>>> >>>>>> Date: Wed, 20 Jul 2016 11:43:27 -0500 >>>>>> From: rodney_bates at lcwb.coop >>>>>> To: jay.krell at cornell.edu; rodney.m.bates at acm.org >>>>>> CC: m3devel at elegosoft.com >>>>>> Subject: Re: [M3devel] purported condition variable problems on Win32? >>>>>> >>>>>> In looking at ThreadWin32.m3 more, I think I see a significant bug. >>>>>> >>>>>> This gets confusing, since there are Modula-3 waits, signals, >>>>>> etc. and similar concepts in Win32, but which are not the >>>>>> same. I am going the try to put "M3-" and "Win-" prefixes on >>>>>> things to make this clearer. I cartainly need to do this >>>>>> for my own benefit, if for nobody else. >>>>>> >>>>>> Suppose two threads are both M3-waiting in Wait(m,c,...), further >>>>>> Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. >>>>>> c.tickets = 0, and c.waiters = 2. >>>>>> >>>>>> Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), >>>>>> which will allow both the waiters to Win-wakeup sometime soon. >>>>>> Before releasing c.lock, Signal increments c.tickets to 1 (supposedly >>>>>> to eventually allow only one waiter to M3-wakeup) and increments >>>>>> c.counter, setting things up so that the waitDone expression at :308 >>>>>> will be TRUE for the waiting threads, the next time they evaluate it. >>>>>> >>>>>> Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, >>>>>> releases c.lock, and tries to acquire m. The problem arises >>>>>> because the second waiter can get to these steps before the first >>>>>> can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The >>>>>> second to reach :308 will find c.tickets # 0, as did the first, >>>>>> and proceed. >>>>>> >>>>>> Each will eventually, do a M3-wake. If this were Posix condition >>>>>> variables, this would be wrong, because Posix says only one >>>>>> waiter proceeds. For M3, it's OK, if unnecessary, because M3 >>>>>> says one or more waiters proceed. However, each will, in the >>>>>> process, get to :322, and DEC(c.tickets). Signal only provided >>>>>> one ticket, but the waiters have taken two. c.tickets goes >>>>>> negative. >>>>>> >>>>>> After this, when some thread can do another (M3-)Wait, and some >>>>>> other does another (M3-)Signal, c.tickets increments only back to >>>>>> zero, which means the new waiter will not M3-wake, even though it >>>>>> should. >>>>>> >>>>>> >>>>>> I am going to try to construct a test case that will force >>>>>> this scenario. >>>>>> >>>>>> See more below: >>>>>> >>>>>> >>>>>> On 07/18/2016 08:53 PM, Jay wrote: >>>>>>> I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. >>>>>>> >>>>>>> Otherwise previous version I think had a very large lock. >>>>>>> >>>>>>> Changing the < to !=, is that really ok? I get that 32bit overflow is possible. >>>>>>> >>>>>>> We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. >>>>>>> >>>>>>> I guess you are saying it would be unfair, which is ok. >>>>>>> >>>>>>> There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. >>>>>>> >>>>>>> I'll check on the unused event. >>>>>>> >>>>>>> - Jay >>>>>>> >>>>>>>> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I spent some time looking over the signal part of ThreadWin32.m3. A few >>>>>>>> comments: >>>>>>>> >>>>>>>> This looks like a very direct implementation of Schmidts' generation count >>>>>>>> algorithm, with a number of important little details additionally taken care >>>>>>>> of. >>>>>>>> >>>>>>>> On small difference is at line 308 (B, below) where Schmidt does >>>>>>>> c.counter>count instead of #. Usually, this would make no difference, >>>>>>>> since counter only increases, but on a long-running program, it could >>>>>>>> overflow. If this happened,as it often does, as silent wrap-around to >>>>>>>> FIRST(INTEGER), some threads could be trapped for an extremely long >>>>>>>> time, waiting for counter come back around greater than their count. As >>>>>>>> coded in ThreadWin32, they would be allowed to proceed normally. >>>>>>>> >>>>>>>> The only glitch would be that, if there were simultaneously waiting threads >>>>>>>> separated by NUMBER(INTEGER) truly different generations (but the same value >>>>>>>> of c.counter), the recent ones could unfairly proceed along with the >>>>>>>> extremely old ones. That seems extremely less likely than the >>>>>>>> mere existence of an overflow. >>>>>>>> >>>>>>>> Both type Condition and type Activation have a field waitEvent: HANDLE. >>>>>>>> Condition.waitEvent is extensively used, but Activation.waitEvent is >>>>>>>> only created at :537 and deleted at :554, but never used in any real >>>>>>>> way. It could be deleted. >>>>>>>> >>>>>>>> As for this code: >>>>>>>> >>>>>>>> EnterCriticalSection(conditionLock); >>>>>>>> >>>>>>>> (* Capture the value of the counter before we start waiting. >>>>>>>> * We will not stop waiting until the counter changes. >>>>>>>> * That is, we will not stop waiting until a signal >>>>>>>> * comes in after we start waiting. >>>>>>>> *) >>>>>>>> >>>>>>>> count := c.counter; >>>>>>>> INC(c.waiters); >>>>>>>> >>>>>>>> LeaveCriticalSection(conditionLock); >>>>>>>> m.release(); (* can this be moved to before taking conditionLock? *) >>>>>>>> >>>>>>>> No, it is important to sample and save c.counter in the order threads wait. >>>>>>>> Retaining m until this is done ensures this. Otherwise, some other thread >>>>>>>> that waited later could have altered c.counter before this one gets conditionLock >>>>>>>> and saves its copy of c.counter. >>>>>>>> >>>>>>>> I have made several comment changes to ThreadWin32.m3 that would have made >>>>>>>> it easier to vet. I will commit these, but only comments. It is either >>>>>>>> very difficult or impossible for me to test or even recompile this module, >>>>>>>> and, especially as fragile as this kind of code is, I don't want to commit >>>>>>>> any substantive changes untested and uncompiled. >>>>>>>> >>>>>>>> More below: >>>>>>>> >>>>>>>>> On 07/07/2016 04:24 AM, Jay K wrote: >>>>>>>>> So...I do NOT understand all of this. >>>>>>>>> >>>>>>>>> >>>>>>>>> However, comparing the three implementations >>>>>>>>> and attempting to understand ours, >>>>>>>>> I am struck by the following >>>>>>>>> >>>>>>>>> - Our broadcast seems ok, but >>>>>>>>> - our signal seems to wake everyone, >>>>>>>>> and then...they race a bit, >>>>>>>>> one will decide it is last, and >>>>>>>>> reset the event, but every prior waiter >>>>>>>>> is still woken. >>>>>>>>> Why not merge the following chunks: >>>>>>>> >>>>>>>> I don't think this will work. It is important that, once a thread has decided, >>>>>>>> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed >>>>>>>> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, >>>>>>>> if it is also the last of its generation to proceed, resets c.waitevent. This >>>>>>>> is ensured by its having already reacquired m first, preventing any other thread >>>>>>>> from returning from its Wait yet. >>>>>> >>>>>> I was wrong about this. There is already no fairness being enforced here. Even with >>>>>> only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is >>>>>> arbitrary anyway. There is no benefit in trying to control the order they reacquire m. >>>>>> c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters >>>>>> that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. >>>>>> >>>>>> Merging, as you proposed, is probably close to the fix for this bug. >>>>>> >>>>>> >>>>>>>> >>>>>>>> And it can't attempt to acquire m while already holding conditionLock, because this would >>>>>>>> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while >>>>>>>> already holding m, at line 265. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> WHILE (NOT alerted) AND (NOT waitDone) DO >>>>>>>>> . >>>>>>>>> . >>>>>>>>> . >>>>>>>>> 1 >>>>>>>>> EnterCriticalSection(conditionLock); >>>>>>>>> waitDone := (c.tickets # 0 AND c.counter # count); >>>>>>>> A:---------^ >>>>>>>> B:--------------------------------------------------^ >>>>>>>>> LeaveCriticalSection(conditionLock); >>>>>>>>> END; (* WHILE *) >>>>>>>>> IF waitDone THEN >>>>>>>>> alerted := FALSE; >>>>>>>>> END; >>>>>>>>> m.acquire(); >>>>>>>>> 2 >>>>>>>>> EnterCriticalSection(conditionLock); >>>>>>>>> DEC(c.waiters); >>>>>>>>> IF waitDone THEN esp. here. >>>>>>>>> DEC(c.tickets); >>>>>>>> >>>>>>>> C:---------------^ >>>>>>>>> lastWaiter := (c.tickets = 0); >>>>>>>>> END; >>>>>>>>> LeaveCriticalSection(conditionLock); >>>>>>>>> >>>>>>>>> >>>>>>>>> That is, if we decrement tickets earlier >>>>>>>>> within the first critical section, >>>>>>>>> while we will still wake everyone, >>>>>>>>> only one will decide waitDone and the rest will keep looping. >>>>>>>>> A downside of this, perhaps, is that waking all for Broadcast >>>>>>>>> might be a little slower. >>>>>>>>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads >>>>>>>>> implementations, they seem to deal with this differently than us, >>>>>>>>> and they each do about the same thing -- they use a counted semaphore. >>>>>>>>> In the boost case, it appears they duplicate the semaphore for every >>>>>>>>> notification generation, which seems expensive. >>>>>>>>> >>>>>>>>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. >>>>>>>>> Or they can do their own reference counting? >>>>>>>>> jdk7 seems to looke like jdk6. >>>>>>>>> The code is gone in jdk8 and I can't easily find the delete in history. >>>>>>>>> The lock merging jdk does probably helps here too. >>>>>>>>> In fact they merge #1 and #2 above as a result. >>>>>>>>> But they still initially wake all threads for signal, not just broadcast. >>>>>>>>> There is also the problem that all the event waits >>>>>>>>> are followed by EnterCriticalSection (or jdk facsimile). >>>>>>>>> - Jay >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> ----! >>>> -----! >>>>>> --! >>>>>>>> --- >>>>>>>>> From: jay.krell at cornell.edu >>>>>>>>> To: m3devel at elegosoft.com >>>>>>>>> Subject: RE: [M3devel] purported condition variable problems on Win32? >>>>>>>>> Date: Tue, 5 Jul 2016 08:26:36 +0000 >>>>>>>>> >>>>>>>>> Here is another implementation to consider: >>>>>>>>> >>>>>>>>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp >>>>>>>>> >>>>>>>>> - Jay >>>>>>>>> >>>>>>>>> >>>>>>>>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> ----! >>>> -----! >>>>>> --! >>>>>>>> --- >>>>>>>>> From: jay.krell at cornell.edu >>>>>>>>> To: m3devel at elegosoft.com >>>>>>>>> Date: Tue, 5 Jul 2016 08:19:23 +0000 >>>>>>>>> Subject: [M3devel] purported condition variable problems on Win32? >>>>>>>>> >>>>>>>>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV >>>>>>>>> >>>>>>>>> vs. >>>>>>>>> >>>>>>>>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >>>>>>>>> vs. >>>>>>>>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain >>>>>>>>> I wrote this: >>>>>>>>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; >>>>>>>>> alertable: BOOLEAN) RAISES {Alerted} = >>>>>>>>> (* LL = m on entry and exit, but not for the duration >>>>>>>>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp >>>>>>>>> * NOTE that they merge the user lock and the condition lock. >>>>>>>>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >>>>>>>>> * "3.3. The Generation Count Solution" >>>>>>>>> *) >>>>>>>>> >>>>>>>>> Do we have the problems described in README.CV? >>>>>>>>> >>>>>>>>> I haven't looked through the ACE code to see >>>>>>>>> to what extent they resemble solution 3.3, or if they >>>>>>>>> changed as a result of this discussion -- which I admit I don't understand >>>>>>>>> and haven't read closely. >>>>>>>>> >>>>>>>>> >>>>>>>>> Spurious wakeups are ok, though should be minimized. >>>>>>>>> >>>>>>>>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> - Jay >>>>>>>>> >>>>>>>>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> M3devel mailing list >>>>>>>>> M3devel at elegosoft.com >>>>>>>>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>>>>>>> >>>>>>>> -- >>>>>>>> Rodney Bates >>>>>>>> rodney.m.bates at acm.org >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Rodney Bates >>>>>> rodney.m.bates at acm.org >>>> >>>> -- >>>> Rodney Bates >>>> rodney.m.bates at acm.org >> >> -- >> Rodney Bates >> rodney.m.bates at acm.org > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Mon Jul 25 18:34:24 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 25 Jul 2016 16:34:24 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> <578FAA2F.6020105@lcwb.coop>, , , <5793DB37.10603@lcwb.coop>, <5794ECAE.5070308@lcwb.coop>, Message-ID: <57963F79.8000906@lcwb.coop> BTW, I see yet a third concurrency bug in ThreadWin32.m3; In AssignSlot, the code to allocate the first slots array, starting at :465 (* make sure we have room to register this guy *) IF slots = NIL THEN LeaveCriticalSection(ADR(slotLock)); slots := NEW (REF ARRAY OF T, 20); EnterCriticalSection(ADR(slotLock)); END; has a race with itself, that needs to be handled the same way that the immediately following code, which enlarges an already-existing slots array, does. (* make sure we have room to register this guy *) IF slots = NIL THEN LeaveCriticalSection(ADR(slotLock)); new_slots := NEW (REF ARRAY OF T, 20); EnterCriticalSection(ADR(slotLock)); IF slots = NIL (* still. *) THEN (* We won any races. *) slots := new_slots; END; END; Otherwise, not only could a different thread have allocated slots after the NIL check and before the assignment to it, it could have put something into its slots, and even enlarged it. Also, both the initial and the enlarging NEW for slots should have an explicit NIL check for out of memory. On 07/25/2016 12:50 AM, Jay K wrote: > btw, I think you are right. > > > In particular, regarding the "directed notify" pattern, > and the fact that we can have a per-thread event. > > > I think you don't see this excact implementation stategy... > you see people creating an event per wait...is because > we have the feature and defect that we require our > own runtime to create our threads, and we don't interoperate > well with threads created otherwise...and there is a fix for this though, > at least on Windows (where we are discussing anyway) via > m3core.dll's DllMain, which we don't implement but probably should. > > > So, let's go ahead and put support in Thread.T. or Thread.Activation > and solve this? > > - Jay > > > -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Tue Jul 26 08:42:31 2016 From: jay.krell at cornell.edu (Jay K) Date: Tue, 26 Jul 2016 06:42:31 -0000 Subject: [M3devel] Grisu vs. NT386 In-Reply-To: References: Message-ID: I think I can fix the nested helper call easily enough in the backend. But note that NT386 is definitely currently broken since the introduction of Grisu. We might also consider having the frontend always form parameters into temporaries. Not clear if it should do this as needed for a few known cases, like LONGINT multiplication, or for all integer and float operations on all types, despite most of them not requiring function calls. Also not clear wrt longer standing helper functions like div/mod. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: Grisu vs. NT386 Date: Mon, 25 Jul 2016 07:27:46 +0000 Fyi, Grisu reveals at least two problems in the NT386 backend.In particular, it uses LONGINT, and we don't have much code that does. The two problem are that NT386: - generates function calls for e.g. 64bit integer muliplication - doesn't like generating function calls within function calls, i.e. VAR a,b: LONGINT; Foo(a * b); It fails an assert -- when it goes to do the multiply, which is a function call, it asserts that no function call is in progress. For now I'll change Grisu: VAR a,b,t: LONGINT; t := a * b; Foo(t); but fixing NT386 shouldn't be difficult. Really, we should allow for: Foo(Bar(a * b) * b); .. the thing is, the frontend handles this -- it produces temporaries for function return results: t1 := Bar(a * b); Foo(t1 * a * b); but it doesn't do so for things like add/subtract/multiply/divide. Perhaps it should.Decent backends, including possibly NT386, will enregister the temporaries and generate good code anyway. Thoughts? The alternative is NT386 needs to maintain a stack where it doesn't previously -- a stack of pending function calls. 2) Grisu uses loophole between LONGREAL and LONGINT. This is reasonable, but perhaps not previously seen.NT386 requires loopholes that convert between integers and floats to be using REAL, at least for one direction.This seem merely historical and it should allow LONGREAL, or in particular, it should require no size change.Historically NT386 backend maintained an internal stack, of 32bit sized things -- variables and constants.I extended that to be variably sized, i.e. 32bits or 64bits, but this code wasn't updated. It should be trivial, maybe just relaxing the assertion. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Jul 28 09:11:07 2016 From: jay.krell at cornell.edu (Jay K) Date: Thu, 28 Jul 2016 07:11:07 -0000 Subject: [M3devel] Grisu vs. NT386 In-Reply-To: References: , Message-ID: This should all be fixed now in the NT386 backend. Verification welcome. Work could be shifted to the frontend for some of this, perhaps. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: RE: Grisu vs. NT386 Date: Tue, 26 Jul 2016 06:42:28 +0000 I think I can fix the nested helper call easily enough in the backend. But note that NT386 is definitely currently broken since the introduction of Grisu. We might also consider having the frontend always form parameters into temporaries. Not clear if it should do this as needed for a few known cases, like LONGINT multiplication, or for all integer and float operations on all types, despite most of them not requiring function calls. Also not clear wrt longer standing helper functions like div/mod. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: Grisu vs. NT386 Date: Mon, 25 Jul 2016 07:27:46 +0000 Fyi, Grisu reveals at least two problems in the NT386 backend.In particular, it uses LONGINT, and we don't have much code that does. The two problem are that NT386: - generates function calls for e.g. 64bit integer muliplication - doesn't like generating function calls within function calls, i.e. VAR a,b: LONGINT; Foo(a * b); It fails an assert -- when it goes to do the multiply, which is a function call, it asserts that no function call is in progress. For now I'll change Grisu: VAR a,b,t: LONGINT; t := a * b; Foo(t); but fixing NT386 shouldn't be difficult. Really, we should allow for: Foo(Bar(a * b) * b); .. the thing is, the frontend handles this -- it produces temporaries for function return results: t1 := Bar(a * b); Foo(t1 * a * b); but it doesn't do so for things like add/subtract/multiply/divide. Perhaps it should.Decent backends, including possibly NT386, will enregister the temporaries and generate good code anyway. Thoughts? The alternative is NT386 needs to maintain a stack where it doesn't previously -- a stack of pending function calls. 2) Grisu uses loophole between LONGREAL and LONGINT. This is reasonable, but perhaps not previously seen.NT386 requires loopholes that convert between integers and floats to be using REAL, at least for one direction.This seem merely historical and it should allow LONGREAL, or in particular, it should require no size change.Historically NT386 backend maintained an internal stack, of 32bit sized things -- variables and constants.I extended that to be variably sized, i.e. 32bits or 64bits, but this code wasn't updated. It should be trivial, maybe just relaxing the assertion. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Thu Jul 28 22:02:10 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Thu, 28 Jul 2016 20:02:10 -0000 Subject: [M3devel] A win32 thread test case Message-ID: <579A646D.3020404@lcwb.coop> I have attached a test case program designed to expose one of the bugs I think I see in ThreadWin32.m3. It runs correctly on AMD64_LINUX. I expect it to fail on Windows, but don't have a Windows machine I can try it on. Anybody willing to try it? It's a self-contained Main module. Just compile and run it. It will announce what it finds. -- Rodney Bates rodney.m.bates at acm.org -------------- next part -------------- (* Test to expose a bug, called the "Twice used ticket" bug, in ThreadWin32.m3: Multiple waiters take multiple tickets when only one was provided by Signal. The result is that the tickets count of a Condition variable goes negative, causing later lost wakeups in Signal. *) MODULE Main ; IMPORT Fmt ; IMPORT Stdio ; IMPORT Thread ; IMPORT Wr ; PROCEDURE W ( Msg : TEXT ) = <* FATAL Thread . Alerted *> <* FATAL Wr . Failure *> BEGIN Wr . PutText ( Stdio . stdout , Msg ) ; Wr . PutText ( Stdio . stdout , Wr . EOL ) ; Wr . Flush ( Stdio . stdout ) END W ; TYPE ThreadNo = [ 0 .. 4 ] ; CONST ThreadNoNull = 0 ; PROCEDURE ThImage ( ThN : ThreadNo ) : TEXT = BEGIN RETURN "Thread " & Fmt . Int ( ThN ) END ThImage (* We want to deterministically control the sequence of thread operations, to create a test case. It is hard to do this without using Condition variables, but we are testing Condition variables, so can't really assume they work correctly. We resort to using a hodge-podge of MUTEXs, busy waiting, and pauses, hoping they are long enough. *) ; CONST BusyWaitInterval = 0.1D0 ; CONST QuiesceInterval = 3.0D0 ; TYPE State = { Null , Idle , Acq , Rel , Wait , Wait2 , Sig } ; TYPE StateSet = SET OF State ; VAR StateMutex : MUTEX ; VAR States := ARRAY ThreadNo OF State { State . Null , .. } (* LL = StateMutex *) ; VAR Holder : ThreadNo := ThreadNoNull (* LL = StateMutex *) ; PROCEDURE StateImage ( St : State ) : TEXT = VAR LResult : TEXT ; BEGIN CASE St OF State . Null => LResult := "null" | State . Idle => LResult := "idle" | State . Acq => LResult := "entering Acquire" | State . Rel => LResult := "entering Release" | State . Wait => LResult := "entering Wait" | State . Wait2 => LResult := "asleep in Wait" | State . Sig => LResult := "entering Signal" END (* CASE *) ; RETURN LResult END StateImage ; EXCEPTION Failure ; VAR ActionMutex : MUTEX ; TYPE ActionProc = PROCEDURE ( ThN : ThreadNo ) ; VAR ActionProcs := ARRAY ThreadNo OF ActionProc { NIL , .. } (* LL = ActionProc *) ; PROCEDURE Action ( ThN : ThreadNo ; Apply : ActionProc ) (* Ask thread number ThN to execute Apply. *) = BEGIN LOCK ActionMutex DO WITH WAct = ActionProcs [ ThN ] DO <* ASSERT WAct = NIL *> WAct := Apply END END ; LOOP (* Wait for test thread to take the direction. *) Thread . Pause ( BusyWaitInterval ) ; LOCK ActionMutex DO IF ActionProcs [ ThN ] = NIL THEN EXIT END (* IF *) END (* LOCK *) END (* LOOP *) END Action ; PROCEDURE ActionWait ( ThN : ThreadNo ; Apply : ActionProc ) RAISES { Failure } (* Ask thread number ThN to execute Apply, expecting Apply to do a Wait(TestMutex, TestCondition), which further implies ThN already holds TestMutex. Note that it waited by its release of TextMutex. *) = BEGIN Action ( ThN , Apply ) ; LOOP (* Wait for ThN to release TestMutex. No other thread is contending for TestMutex, so this is an indication that ThN has stopped inside Wait. *) LOCK TestMutex DO LOCK StateMutex DO WITH WSt = States [ ThN ] DO IF WSt # State . Wait THEN W ( ThImage ( ThN ) & " Failed to wait in Wait." ) ; RAISE Failure ELSE <* ASSERT Holder = ThN *> Holder := ThreadNoNull ; WSt := State . Wait2 ; W ( ThImage ( ThN ) & " Waiting." ) ; EXIT END (* IF *) END (* WITH *) END (* LOCK StateMutex *) END (* LOCK TestMutex*) END (* LOOP *) END ActionWait ; PROCEDURE WaitForHeld ( ) : ThreadNo (* Busy wait until some thread holds TestMutex, then return its ThreadNo. *) = VAR LHolder : ThreadNo ; BEGIN LOOP Thread . Pause ( BusyWaitInterval ) ; LOCK StateMutex DO LHolder := Holder END (* LOCK *) ; IF LHolder # ThreadNoNull THEN EXIT END END (* LOOP *) ; RETURN LHolder END WaitForHeld ; PROCEDURE WaitForStateSet ( ThN : ThreadNo ; Sts : StateSet ) (* Busy wait for thread number ThN to get into one of Sts. *) = BEGIN LOOP LOCK StateMutex DO WITH WSt = States [ ThN ] DO IF WSt IN Sts THEN (* This is what we want. *) EXIT END (* IF *) END (* WITH *) END (* LOCK *) ; Thread . Pause ( BusyWaitInterval ) END (* LOOP *) END WaitForStateSet ; PROCEDURE WaitForState ( ThN : ThreadNo ; St : State ) (* Busy wait for action thread to get into St. *) = BEGIN WaitForStateSet ( ThN , StateSet { St } ) END WaitForState ; PROCEDURE NoteWhetherStateSet ( ThN : ThreadNo ; Sts : StateSet ; YesMsg , NoMsg := "" ) : BOOLEAN (* Write whether thread number ThN is in one of Sts and return it. *) = VAR LResult : BOOLEAN ; VAR LState : State ; VAR LMsg : TEXT ; BEGIN LOCK StateMutex DO LState := States [ ThN ] ; LResult := LState IN Sts ; LMsg := ThImage ( ThN ) & " is " & StateImage ( LState ) ; IF LResult THEN IF YesMsg # NIL THEN LMsg := LMsg & YesMsg END ELSE IF NoMsg # NIL THEN LMsg := LMsg & NoMsg END END (* IF *) ; W ( LMsg ) END (* LOCK *) ; RETURN LResult END NoteWhetherStateSet ; PROCEDURE NoteWhetherState ( ThN : ThreadNo ; St : State ; YesMsg , NoMsg := "" ) : BOOLEAN (* Write whether thread number ThN is in St and return it. *) = BEGIN RETURN NoteWhetherStateSet ( ThN , StateSet { St } , YesMsg , NoMsg ) END NoteWhetherState ; VAR TestMutex : MUTEX ; VAR TestCond : Thread . Condition ; VAR Threads : ARRAY ThreadNo OF Thread . T ; TYPE Cl = Thread . Closure OBJECT ClThN : ThreadNo OVERRIDES apply := TestApply END ; VAR Closures : ARRAY ThreadNo OF Cl ; PROCEDURE TestApply ( Self : Cl ) : REFANY (* Loop for threads that execute test actions on Mutexs and Conditions. *) = VAR LProc : ActionProc ; BEGIN LOOP (* Busy wait for work. *) Thread . Pause ( 0.2D0 (* Not too extremely busy. *) + FLOAT ( Self . ClThN , LONGREAL ) * 0.01D0 (* ^Avoid staying in phase. *) ) ; LOCK ActionMutex DO WITH WProc = ActionProcs [ Self . ClThN ] DO IF WProc # NIL THEN (* We have some work to do. *) LProc := WProc ; WProc := NIL END (* IF *) END (* WITH *) END (* LOCK *) ; IF LProc # NIL THEN LProc ( Self . ClThN ) (* ^May or may not return right away. *) ; LProc := NIL END (* IF *) END (* LOOP *) ; <* NOWARN *> (* Intentionally unreachable. *) RETURN NIL END TestApply ; PROCEDURE DoAcq ( ThN : ThreadNo ) = BEGIN LOCK StateMutex DO States [ ThN ] := State . Acq END ; W ( ThImage ( ThN ) & " Entering Acquire of TestMutex." ) ; Thread . Acquire ( TestMutex ) ; W ( ThImage ( ThN ) & " Acquired TestMutex." ) ; LOCK StateMutex DO WITH WThN = States [ ThN ] DO <* ASSERT WThN = State . Acq *> WThN := State . Idle ; <* ASSERT Holder = ThreadNoNull *> Holder := ThN END (* WITH *) END (* LOCK *) END DoAcq ; PROCEDURE DoRel ( ThN : ThreadNo ) = BEGIN LOCK StateMutex DO States [ ThN ] := State . Rel END ; W ( ThImage ( ThN ) & " Entering release of TestMutex." ) ; Thread . Release ( TestMutex ) ; W ( ThImage ( ThN ) & " Released TestMutex." ) ; LOCK StateMutex DO WITH WThN = States [ ThN ] DO <* ASSERT WThN = State . Rel *> WThN := State . Idle ; <* ASSERT Holder = ThN *> Holder := ThreadNoNull END (* WITH *) END (* LOCK *) END DoRel ; PROCEDURE DoWait ( ThN : ThreadNo ) = BEGIN LOCK StateMutex DO WITH WThN = States [ ThN ] DO <* ASSERT Holder = ThN *> <* ASSERT WThN = State . Idle *> WThN := State . Wait END (* WITH *) END (* LOCK *) ; W ( ThImage ( ThN ) & " Entering Wait on TestCond." ) ; Thread . Wait ( TestMutex , TestCond ) ; W ( ThImage ( ThN ) & " Was Signalled in TestCond and reacquired TestMutex." ) ; LOCK StateMutex DO WITH WThN = States [ ThN ] DO <* ASSERT WThN = State . Wait2 *> States [ ThN ] := State . Idle ; Holder := ThN END (* WITH *) END (* LOCK *) END DoWait ; PROCEDURE DoSignal ( ThN : ThreadNo ) = BEGIN LOCK StateMutex DO States [ ThN ] := State . Sig END ; W ( ThImage ( ThN ) & " Entering Signal on TestCond" ) ; Thread . Signal ( TestCond ) ; W ( ThImage ( ThN ) & " TestCond Signalled " ) ; LOCK StateMutex DO States [ ThN ] := State . Idle END END DoSignal ; PROCEDURE ForceSignalled ( ThN : ThreadNo ) (* PRE: Only ThN could hold TestMutex. If thread ThN is asleep in TestCondition waiting for a Signal, Signal it and let it reacquire TestMutex. Either way, finally make it release TestMutex. *) = VAR LThN : ThreadNo ; BEGIN (* Allow ThN time to acquire TestMutex, if it has been signalled. *) Thread . Pause ( QuiesceInterval ) ; IF NoteWhetherState ( ThN , State . Wait2 ) THEN Action ( 3 , DoSignal ) ; WaitForState ( 3 , State . Idle ) ; LThN := WaitForHeld ( ) ; <* ASSERT LThN = ThN *> END (* IF *) ; Action ( ThN , DoRel ) ; WaitForState ( ThN , State . Idle ) END ForceSignalled ; PROCEDURE TestSeq ( ) = VAR LThNo : ThreadNo ; BEGIN TRY (* Thread 1 waits: *) Action ( 1 , DoAcq ) ; WaitForState ( 1 , State . Idle ) ; ActionWait ( 1 , DoWait ) ; WaitForState ( 1 , State . Wait2 ) (* Thread 2 waits: *) ; Action ( 2 , DoAcq ) ; WaitForState ( 2 , State . Idle ) ; ActionWait ( 2 , DoWait ) ; WaitForState ( 2 , State . Wait2 ) (* Here, there are two waiters. *) (* The Wait implementation must unblock at least one of the waiting threads, but is allowed to unblock up to all of them. The Posix and PThread implementations unblock only one. The buggy Win32 implementation unblocks both in unpredictable order. We want to get both unblocked from inside Wait and also let them acquire and release TestMutex, to set up consistently, in preparation for testing for the actual bug. *) (* Hold TestMutex to limit how far 1 and 2 can get after Wait unblocks them. *) ; Action ( 3 , DoAcq ) ; WaitForState ( 3 , State . Idle ) (* Signal. Each of 1 & 2, when unblocked, will block again trying to reacquire TestMutex.*) ; Action ( 3 , DoSignal ) ; WaitForState ( 3 , State . Idle ) (* Now release TestMutex, so the first thread unblocked inside Wait by the Signal will acquire it. *) ; Action ( 3 , DoRel ) ; WaitForState ( 3 , State . Idle ) (* See what thread was unblocked. *) ; LThNo := WaitForHeld ( ) ; IF LThNo = 1 THEN (* 1 holds TestMutex. 2 will be asleep, either in TestCondition or for TestMutex. *) Action ( 1 , DoRel ) ; WaitForState ( 1 , State . Idle ) ; ForceSignalled ( 2 ) ELSE (* 1 has not gotten out of Wait. 2 Must have done so. *) <* ASSERT LThNo = 2 *> Action ( 2 , DoRel ) ; WaitForState ( 2 , State . Idle ) ; ForceSignalled ( 1 ) END (* IF *) (* Tread 4 Waits. *) ; Action ( 4 , DoAcq ) ; WaitForState ( 4 , State . Idle ) ; ActionWait ( 4 , DoWait ) ; WaitForState ( 4 , State . Wait2 ) (* Thread 3 Signals. *) ; Action ( 3 , DoAcq ) ; WaitForState ( 3 , State . Idle ) ; Action ( 3 , DoSignal ) ; WaitForState ( 3 , State . Idle ) ; Action ( 3 , DoRel ) ; WaitForState ( 3 , State . Idle ) (* Give 4 time to reacquire TestMutex, if it was signalled *) ; Thread . Pause ( QuiesceInterval ) (* See whether 4 got signalled. *) ; WaitForStateSet ( 4 , StateSet { State . Wait2 , State . Idle } ) ; IF NoteWhetherState ( 4 , State . Idle ) THEN Action ( 4 , DoRel ) ; WaitForState ( 4 , State . Idle ) ; W ( "SUCCESS: all as expected.") ELSE W ( "FAILURE: This is the \"twice used ticket\" bug we are testing for." ) ; RAISE Failure END (* IF *) EXCEPT | Failure => END (* EXCEPT *) END TestSeq ; PROCEDURE Init ( ) = BEGIN ActionMutex := NEW ( MUTEX ) ; StateMutex := NEW ( MUTEX ) ; TestMutex := NEW ( MUTEX ) ; TestCond := NEW ( Thread . Condition ) ; FOR LThNo := FIRST ( ThreadNo ) TO LAST ( ThreadNo ) DO Closures [ LThNo ] := NEW ( Cl , ClThN := LThNo ) ; States [ LThNo ] := State . Idle (* Waiting for directions. *) ; Threads [ LThNo ] := Thread . Fork ( Closures [ LThNo ] ) END (* FOR *) END Init ; BEGIN Init ( ) ; TestSeq ( ) END Main . From jay.krell at cornell.edu Sat Jul 30 23:18:15 2016 From: jay.krell at cornell.edu (Jay K) Date: Sat, 30 Jul 2016 21:18:15 -0000 Subject: [M3devel] A win32 thread test case In-Reply-To: <579A646D.3020404@lcwb.coop> References: <579A646D.3020404@lcwb.coop> Message-ID: I fixed the new_slots problem. I agree there are problems here. I think Java got away with a similar implementation because their interface is a bit different. i.e. the lock and condition are merged, and/or notification requires the caller to take the lock. We can't do this. We allow signal/broadcast w/o holding a lock. We'd have to resort to a global lock I guess. I suspect nobody else uses this solution. I believe we can solve it better and worse than others. Most libraries do not have their own "CreateThread" function.That is, most libraries let you call pthread_create or Win32 CreateThread,and the library will interoperate with any thread that comes its way.And most libraries don't seem to use thread locals in their solution.By "most libraries", I mean the pthreads on win32 package and Boost. Modula-3 isn't clearly this way. This isn't great, but it is the currentsituation. I believe it is fixable.That is, Modula-3 I believe requires using its library to create threads.This is not great for interoperation. You want to be able to be usedby code that isn't away of you, that just creates threads in the "normal" wayfor their platform. And then, either the current state, or a fix I have in mind, can be takenadvantage of to do "directed notify" w/o creating a kernel eventper wait or notify, like other solutions do. As to the fix, to not require threads go through our "CreateThread",I believe on Windows (which is all that is relevant here), we should havea DllMain that allocates thread locals. We can only easily have a DllMain if we are a .dll.Or we can use __declspec(thread).__declspec(thread) works if the .exe is statically linked to the exe or the dll,or on Vista and newer, but not in a .dll loaded by LoadLibrary prior to Vista.Perhaps that is good enough. If we require Vista then we can just drop a bunch of our code and usethe Win32 condition variables presumably.I remain curious how the condition variables work there, i.e. we can'timplement them ourselves, but it is possible they either require kernelsupport or are well beyond our abilities. - Jay > Date: Thu, 28 Jul 2016 15:00:45 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: A win32 thread test case > > I have attached a test case program designed to expose one of the bugs > I think I see in ThreadWin32.m3. It runs correctly on AMD64_LINUX. > I expect it to fail on Windows, but don't have a Windows machine I > can try it on. > > Anybody willing to try it? It's a self-contained Main module. Just > compile and run it. It will announce what it finds. > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 00:15:28 2016 From: jay.krell at cornell.edu (Jay K) Date: Sat, 30 Jul 2016 22:15:28 -0000 Subject: [M3devel] why thread slots? Message-ID: What is the point of all this "slot" stuff in the thread library? It seems kind of self-referential and pointless. I might try removing it and see what the results are. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 02:47:27 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 00:47:27 -0000 Subject: [M3devel] A win32 thread test case In-Reply-To: References: <579A646D.3020404@lcwb.coop>, Message-ID: ThreadPThread.m3 had the same narrow startup race condition. I fixed it.Again I'm not seeing commit mails. On the larger matter: I think there is a solution, in that, if you look atThreadPThread.m3 and how it uses Posix condition variables,in that it does a fair amount of the work itself,you can term this "condition within condition". So I think maybe we can use the counting solutionand 1) merge the locks and/or 2) take the lock in Signal/Broadcast. But I have to look at it more. Specifically, in ThreadPThread.m3: Broadcast: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; next := t.nextWaiter; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Signal: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Becaise they signal while locked, we can implement this limitedform of condition variable, and then use the Modula-3 pthread logicusing that. Make some sense? Essentially we will have almost-posix condition variables,and exactly-posix mutexes, and then we can use just the pthread codeon top of that. Almost-posix condition variable is a posix condition variable, exceptit is guaranteed you will call pthread_cond_signal with the samelock held as when you call pthread_cond_wait. On a pthread system, the almost-posix condition variables and exactly-posixmutexes are just pass through to the underlying pthreads. On a win32 system, an almost-posix condition variable is similarto what we have now, except, because we know this lock is held,"XWait" can remove some of the leave/enter cycles and remove race conditions. Essentially we only need a "monitor", which is one lock and one condition variable. OR, we can use "directed notification" where each thread has an event, and threads are linked through a "wait block" i.e. the condition. This should work well too. It depends on us owing CreateThread, which we already do. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sat, 30 Jul 2016 21:18:08 +0000 I fixed the new_slots problem. I agree there are problems here. I think Java got away with a similar implementation because their interface is a bit different. i.e. the lock and condition are merged, and/or notification requires the caller to take the lock. We can't do this. We allow signal/broadcast w/o holding a lock. We'd have to resort to a global lock I guess. I suspect nobody else uses this solution. I believe we can solve it better and worse than others. Most libraries do not have their own "CreateThread" function.That is, most libraries let you call pthread_create or Win32 CreateThread,and the library will interoperate with any thread that comes its way.And most libraries don't seem to use thread locals in their solution.By "most libraries", I mean the pthreads on win32 package and Boost. Modula-3 isn't clearly this way. This isn't great, but it is the currentsituation. I believe it is fixable.That is, Modula-3 I believe requires using its library to create threads.This is not great for interoperation. You want to be able to be usedby code that isn't away of you, that just creates threads in the "normal" wayfor their platform. And then, either the current state, or a fix I have in mind, can be takenadvantage of to do "directed notify" w/o creating a kernel eventper wait or notify, like other solutions do. As to the fix, to not require threads go through our "CreateThread",I believe on Windows (which is all that is relevant here), we should havea DllMain that allocates thread locals. We can only easily have a DllMain if we are a .dll.Or we can use __declspec(thread).__declspec(thread) works if the .exe is statically linked to the exe or the dll,or on Vista and newer, but not in a .dll loaded by LoadLibrary prior to Vista.Perhaps that is good enough. If we require Vista then we can just drop a bunch of our code and usethe Win32 condition variables presumably.I remain curious how the condition variables work there, i.e. we can'timplement them ourselves, but it is possible they either require kernelsupport or are well beyond our abilities. - Jay > Date: Thu, 28 Jul 2016 15:00:45 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: A win32 thread test case > > I have attached a test case program designed to expose one of the bugs > I think I see in ThreadWin32.m3. It runs correctly on AMD64_LINUX. > I expect it to fail on Windows, but don't have a Windows machine I > can try it on. > > Anybody willing to try it? It's a self-contained Main module. Just > compile and run it. It will announce what it finds. > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 04:52:05 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 02:52:05 -0000 Subject: [M3devel] A win32 thread test case In-Reply-To: References: <579A646D.3020404@lcwb.coop>, , Message-ID: While I agree this is confusing, it is holding together for me and I have mostly coded it up. Repeating myself: In this scheme there are two types of condition variable. There are Modula-3 condition variables and posix-like/schmidt-like condition variables. The posix-like/smidt-like condition variables: Can be implemented directly via posix condition variables (if Windows pre-Vista had them). They are implemented as a slight modification of the Smidt counter method. The differences are that condition.lock is removed. Caller of cond_signal/cond_broadcast is responsible to hold the same lock as he holds when calling cond_wait. Some of the lock/unlock cycles are removed, and I hope therefore all the bugs are fixed. I still have to work through your examples. Modula-3 condition variables are then built on top of pthread-like/schmidt condition variables just like ThreadPThread.m3. This includes "alertable". The separate event previously in the Win32 implementation becomes a boolean plus cond_signal.XPause changes to call XWait, essentially. I was thinking we could reuse the Win32 thread alerting feature, but this alerts for other reasons so maybe isn't approprirate. If this works, we should be able to share more code between the pthread and win32 implementations.We just need a name for this slightly constrained pthread interface, i.e. where the condition variables assume the lock is held by the caller of signal/broadcast (we don't use broadcast). - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 00:47:24 +0000 ThreadPThread.m3 had the same narrow startup race condition. I fixed it.Again I'm not seeing commit mails. On the larger matter: I think there is a solution, in that, if you look atThreadPThread.m3 and how it uses Posix condition variables,in that it does a fair amount of the work itself,you can term this "condition within condition". So I think maybe we can use the counting solutionand 1) merge the locks and/or 2) take the lock in Signal/Broadcast. But I have to look at it more. Specifically, in ThreadPThread.m3: Broadcast: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; next := t.nextWaiter; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Signal: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Becaise they signal while locked, we can implement this limitedform of condition variable, and then use the Modula-3 pthread logicusing that. Make some sense? Essentially we will have almost-posix condition variables,and exactly-posix mutexes, and then we can use just the pthread codeon top of that. Almost-posix condition variable is a posix condition variable, exceptit is guaranteed you will call pthread_cond_signal with the samelock held as when you call pthread_cond_wait. On a pthread system, the almost-posix condition variables and exactly-posixmutexes are just pass through to the underlying pthreads. On a win32 system, an almost-posix condition variable is similarto what we have now, except, because we know this lock is held,"XWait" can remove some of the leave/enter cycles and remove race conditions. Essentially we only need a "monitor", which is one lock and one condition variable. OR, we can use "directed notification" where each thread has an event, and threads are linked through a "wait block" i.e. the condition. This should work well too. It depends on us owing CreateThread, which we already do. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sat, 30 Jul 2016 21:18:08 +0000 I fixed the new_slots problem. I agree there are problems here. I think Java got away with a similar implementation because their interface is a bit different. i.e. the lock and condition are merged, and/or notification requires the caller to take the lock. We can't do this. We allow signal/broadcast w/o holding a lock. We'd have to resort to a global lock I guess. I suspect nobody else uses this solution. I believe we can solve it better and worse than others. Most libraries do not have their own "CreateThread" function.That is, most libraries let you call pthread_create or Win32 CreateThread,and the library will interoperate with any thread that comes its way.And most libraries don't seem to use thread locals in their solution.By "most libraries", I mean the pthreads on win32 package and Boost. Modula-3 isn't clearly this way. This isn't great, but it is the currentsituation. I believe it is fixable.That is, Modula-3 I believe requires using its library to create threads.This is not great for interoperation. You want to be able to be usedby code that isn't away of you, that just creates threads in the "normal" wayfor their platform. And then, either the current state, or a fix I have in mind, can be takenadvantage of to do "directed notify" w/o creating a kernel eventper wait or notify, like other solutions do. As to the fix, to not require threads go through our "CreateThread",I believe on Windows (which is all that is relevant here), we should havea DllMain that allocates thread locals. We can only easily have a DllMain if we are a .dll.Or we can use __declspec(thread).__declspec(thread) works if the .exe is statically linked to the exe or the dll,or on Vista and newer, but not in a .dll loaded by LoadLibrary prior to Vista.Perhaps that is good enough. If we require Vista then we can just drop a bunch of our code and usethe Win32 condition variables presumably.I remain curious how the condition variables work there, i.e. we can'timplement them ourselves, but it is possible they either require kernelsupport or are well beyond our abilities. - Jay > Date: Thu, 28 Jul 2016 15:00:45 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: A win32 thread test case > > I have attached a test case program designed to expose one of the bugs > I think I see in ThreadWin32.m3. It runs correctly on AMD64_LINUX. > I expect it to fail on Windows, but don't have a Windows machine I > can try it on. > > Anybody willing to try it? It's a self-contained Main module. Just > compile and run it. It will announce what it finds. > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 05:54:58 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 03:54:58 -0000 Subject: [M3devel] joinMu global? Message-ID: Why is joinMu global, instead of e.g. per thread? This presumably is a scalability blocker? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 06:14:48 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 04:14:48 -0000 Subject: [M3devel] joinMu global? In-Reply-To: References: Message-ID: I guess it is probably ok in the scheme of things, since creating threads is also fairly serialized.:( - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: joinMu global? Date: Sun, 31 Jul 2016 03:54:51 +0000 Why is joinMu global, instead of e.g. per thread? This presumably is a scalability blocker? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 06:24:18 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 04:24:18 -0000 Subject: [M3devel] A win32 thread test case In-Reply-To: References: <579A646D.3020404@lcwb.coop>, , , Message-ID: Something like attached?Still testing..might be crashing.A lot of the code and patterns come from ThreadPThread.m3. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 02:52:02 +0000 While I agree this is confusing, it is holding together for me and I have mostly coded it up. Repeating myself: In this scheme there are two types of condition variable. There are Modula-3 condition variables and posix-like/schmidt-like condition variables. The posix-like/smidt-like condition variables: Can be implemented directly via posix condition variables (if Windows pre-Vista had them). They are implemented as a slight modification of the Smidt counter method. The differences are that condition.lock is removed. Caller of cond_signal/cond_broadcast is responsible to hold the same lock as he holds when calling cond_wait. Some of the lock/unlock cycles are removed, and I hope therefore all the bugs are fixed. I still have to work through your examples. Modula-3 condition variables are then built on top of pthread-like/schmidt condition variables just like ThreadPThread.m3. This includes "alertable". The separate event previously in the Win32 implementation becomes a boolean plus cond_signal.XPause changes to call XWait, essentially. I was thinking we could reuse the Win32 thread alerting feature, but this alerts for other reasons so maybe isn't approprirate. If this works, we should be able to share more code between the pthread and win32 implementations.We just need a name for this slightly constrained pthread interface, i.e. where the condition variables assume the lock is held by the caller of signal/broadcast (we don't use broadcast). - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 00:47:24 +0000 ThreadPThread.m3 had the same narrow startup race condition. I fixed it.Again I'm not seeing commit mails. On the larger matter: I think there is a solution, in that, if you look atThreadPThread.m3 and how it uses Posix condition variables,in that it does a fair amount of the work itself,you can term this "condition within condition". So I think maybe we can use the counting solutionand 1) merge the locks and/or 2) take the lock in Signal/Broadcast. But I have to look at it more. Specifically, in ThreadPThread.m3: Broadcast: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; next := t.nextWaiter; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Signal: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Becaise they signal while locked, we can implement this limitedform of condition variable, and then use the Modula-3 pthread logicusing that. Make some sense? Essentially we will have almost-posix condition variables,and exactly-posix mutexes, and then we can use just the pthread codeon top of that. Almost-posix condition variable is a posix condition variable, exceptit is guaranteed you will call pthread_cond_signal with the samelock held as when you call pthread_cond_wait. On a pthread system, the almost-posix condition variables and exactly-posixmutexes are just pass through to the underlying pthreads. On a win32 system, an almost-posix condition variable is similarto what we have now, except, because we know this lock is held,"XWait" can remove some of the leave/enter cycles and remove race conditions. Essentially we only need a "monitor", which is one lock and one condition variable. OR, we can use "directed notification" where each thread has an event, and threads are linked through a "wait block" i.e. the condition. This should work well too. It depends on us owing CreateThread, which we already do. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sat, 30 Jul 2016 21:18:08 +0000 I fixed the new_slots problem. I agree there are problems here. I think Java got away with a similar implementation because their interface is a bit different. i.e. the lock and condition are merged, and/or notification requires the caller to take the lock. We can't do this. We allow signal/broadcast w/o holding a lock. We'd have to resort to a global lock I guess. I suspect nobody else uses this solution. I believe we can solve it better and worse than others. Most libraries do not have their own "CreateThread" function.That is, most libraries let you call pthread_create or Win32 CreateThread,and the library will interoperate with any thread that comes its way.And most libraries don't seem to use thread locals in their solution.By "most libraries", I mean the pthreads on win32 package and Boost. Modula-3 isn't clearly this way. This isn't great, but it is the currentsituation. I believe it is fixable.That is, Modula-3 I believe requires using its library to create threads.This is not great for interoperation. You want to be able to be usedby code that isn't away of you, that just creates threads in the "normal" wayfor their platform. And then, either the current state, or a fix I have in mind, can be takenadvantage of to do "directed notify" w/o creating a kernel eventper wait or notify, like other solutions do. As to the fix, to not require threads go through our "CreateThread",I believe on Windows (which is all that is relevant here), we should havea DllMain that allocates thread locals. We can only easily have a DllMain if we are a .dll.Or we can use __declspec(thread).__declspec(thread) works if the .exe is statically linked to the exe or the dll,or on Vista and newer, but not in a .dll loaded by LoadLibrary prior to Vista.Perhaps that is good enough. If we require Vista then we can just drop a bunch of our code and usethe Win32 condition variables presumably.I remain curious how the condition variables work there, i.e. we can'timplement them ourselves, but it is possible they either require kernelsupport or are well beyond our abilities. - Jay > Date: Thu, 28 Jul 2016 15:00:45 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: A win32 thread test case > > I have attached a test case program designed to expose one of the bugs > I think I see in ThreadWin32.m3. It runs correctly on AMD64_LINUX. > I expect it to fail on Windows, but don't have a Windows machine I > can try it on. > > Anybody willing to try it? It's a self-contained Main module. Just > compile and run it. It will announce what it finds. > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ThreadWin32.m3 Type: application/octet-stream Size: 45813 bytes Desc: not available URL: From jay.krell at cornell.edu Sun Jul 31 10:19:55 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 08:19:55 -0000 Subject: [M3devel] Grisu vs. NT386 In-Reply-To: References: , , Message-ID: nope; it compiles but still doesn't workI'll disable it for now, on NT386 (which isn't quite the right condition, since the C backend works there and likely works for Grisu) - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: RE: Grisu vs. NT386 Date: Thu, 28 Jul 2016 07:11:04 +0000 This should all be fixed now in the NT386 backend. Verification welcome. Work could be shifted to the frontend for some of this, perhaps. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: RE: Grisu vs. NT386 Date: Tue, 26 Jul 2016 06:42:28 +0000 I think I can fix the nested helper call easily enough in the backend. But note that NT386 is definitely currently broken since the introduction of Grisu. We might also consider having the frontend always form parameters into temporaries. Not clear if it should do this as needed for a few known cases, like LONGINT multiplication, or for all integer and float operations on all types, despite most of them not requiring function calls. Also not clear wrt longer standing helper functions like div/mod. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: Grisu vs. NT386 Date: Mon, 25 Jul 2016 07:27:46 +0000 Fyi, Grisu reveals at least two problems in the NT386 backend.In particular, it uses LONGINT, and we don't have much code that does. The two problem are that NT386: - generates function calls for e.g. 64bit integer muliplication - doesn't like generating function calls within function calls, i.e. VAR a,b: LONGINT; Foo(a * b); It fails an assert -- when it goes to do the multiply, which is a function call, it asserts that no function call is in progress. For now I'll change Grisu: VAR a,b,t: LONGINT; t := a * b; Foo(t); but fixing NT386 shouldn't be difficult. Really, we should allow for: Foo(Bar(a * b) * b); .. the thing is, the frontend handles this -- it produces temporaries for function return results: t1 := Bar(a * b); Foo(t1 * a * b); but it doesn't do so for things like add/subtract/multiply/divide. Perhaps it should.Decent backends, including possibly NT386, will enregister the temporaries and generate good code anyway. Thoughts? The alternative is NT386 needs to maintain a stack where it doesn't previously -- a stack of pending function calls. 2) Grisu uses loophole between LONGREAL and LONGINT. This is reasonable, but perhaps not previously seen.NT386 requires loopholes that convert between integers and floats to be using REAL, at least for one direction.This seem merely historical and it should allow LONGREAL, or in particular, it should require no size change.Historically NT386 backend maintained an internal stack, of 32bit sized things -- variables and constants.I extended that to be variably sized, i.e. 32bits or 64bits, but this code wasn't updated. It should be trivial, maybe just relaxing the assertion. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 10:25:57 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 08:25:57 -0000 Subject: [M3devel] using interface compiler in m3core/Grisu? Message-ID: Why can't I say this? C:\dev2\cm3.2\m3-libs\m3core\src\float\Common\grisu\Grisu.m3 IMPORT Compiler; PROCEDURE FastDtoa(v : LONGREAL;.. (* temporarily disable for NT386 until problems investigated *) IF Compiler.ThisPlatform = Compiler.Platform.NT386 THEN RETURN FALSE END; new source -> compiling Grisu.m3 Fatal Error: bad version stamps: Grisu.m3 missing export: Compiler.Platform imported by Grisu.m3missing export: Compiler.ThisPlatform imported by Grisu.m3 *** execution of [, ] failed *** ? I know how to workaround in an ugly way -- via m3makefile chosing a different file.. Thank you, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 10:41:00 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 08:41:00 -0000 Subject: [M3devel] using interface compiler in m3core/Grisu? In-Reply-To: References: Message-ID: oh I see, the documentation includes but I removed it years ago as it is kind of a maintenance burden. Kinda. Author: Jay Michael Krell 2010-12-30 05:35:01Committer: Jay Michael Krell 2010-12-30 05:35:01Parent: 1e492adb4f0b0676b3d62aec23da45d197907302 (add OpenBSD/alpha here too)Child: 655eeacc2d2cbe92676cb95840169c81ede4d0e7 (Merge Unicode branch into head. For a summary of the changes, see)Branches: master, remotes/origin/devel_unicode, remotes/origin/masterFollows: Precedes: verification-release, verification-release-x86_64 remove Compiler.ThisPlatform yet another piece of porting work gone There are no uses of this, and any uses would be somewhat dubious, and easy to replace with a compile time check of TARGET in Quake code. See this code for how: - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: using interface compiler in m3core/Grisu? Date: Sun, 31 Jul 2016 08:25:54 +0000 Why can't I say this? C:\dev2\cm3.2\m3-libs\m3core\src\float\Common\grisu\Grisu.m3 IMPORT Compiler; PROCEDURE FastDtoa(v : LONGREAL;.. (* temporarily disable for NT386 until problems investigated *) IF Compiler.ThisPlatform = Compiler.Platform.NT386 THEN RETURN FALSE END; new source -> compiling Grisu.m3 Fatal Error: bad version stamps: Grisu.m3 missing export: Compiler.Platform imported by Grisu.m3missing export: Compiler.ThisPlatform imported by Grisu.m3 *** execution of [, ] failed *** ? I know how to workaround in an ugly way -- via m3makefile chosing a different file.. Thank you, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 11:07:12 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 09:07:12 -0000 Subject: [M3devel] idioms for tracking initialization state and raising errors? Message-ID: Is this a correct idiom?If so, I think it is fairly reasonable. That is: be very willing to "early return" put all cleanup in finally raise errors in finally In particular, I don't want to repeat the cleanup.I don't want to raise before leaving critical sections. I don't intend this to raise within a raise, but onlyfrom the "normal" exit of the finally block. I also don't want extra local booleans, i.e. raise: boolean to indicate failure.Just the one to indicate success. PROCEDURE InitMutex (mutex: Mutex) = VAR lock: PCRITICAL_SECTION := NIL; locked: PCRITICAL_SECTION := NIL; BEGIN IF mutex.initialized THEN RETURN END; TRY lock := NewCriticalSection(); IF lock = NIL THEN RETURN; END; EnterCriticalSection(ADR(initLock)); locked := ADR(initLock); IF mutex.initialized THEN RETURN END; (* We won the race. *) RTHeapRep.RegisterFinalCleanup (mutex, CleanMutex); mutex.lock := lock; lock := NIL; mutex.initialized := TRUE; FINALLY IF locked # NIL THEN LeaveCriticalSection(locked); END; DelCriticalSection(lock); IF NOT mutex.initialized THEN (* Raise after leaving critical section. *) RuntimeError.Raise (RuntimeError.T.OutOfMemory); END; END; END InitMutex; Thank you, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 12:37:34 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 10:37:34 -0000 Subject: [M3devel] A win32 thread test case In-Reply-To: References: <579A646D.3020404@lcwb.coop>, , , , Message-ID: a fallacy here is that..broadcast isn't used, only signal, so the implementation is more general than needed.An even per thread should work. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 04:24:14 +0000 Something like attached?Still testing..might be crashing.A lot of the code and patterns come from ThreadPThread.m3. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 02:52:02 +0000 While I agree this is confusing, it is holding together for me and I have mostly coded it up. Repeating myself: In this scheme there are two types of condition variable. There are Modula-3 condition variables and posix-like/schmidt-like condition variables. The posix-like/smidt-like condition variables: Can be implemented directly via posix condition variables (if Windows pre-Vista had them). They are implemented as a slight modification of the Smidt counter method. The differences are that condition.lock is removed. Caller of cond_signal/cond_broadcast is responsible to hold the same lock as he holds when calling cond_wait. Some of the lock/unlock cycles are removed, and I hope therefore all the bugs are fixed. I still have to work through your examples. Modula-3 condition variables are then built on top of pthread-like/schmidt condition variables just like ThreadPThread.m3. This includes "alertable". The separate event previously in the Win32 implementation becomes a boolean plus cond_signal.XPause changes to call XWait, essentially. I was thinking we could reuse the Win32 thread alerting feature, but this alerts for other reasons so maybe isn't approprirate. If this works, we should be able to share more code between the pthread and win32 implementations.We just need a name for this slightly constrained pthread interface, i.e. where the condition variables assume the lock is held by the caller of signal/broadcast (we don't use broadcast). - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 00:47:24 +0000 ThreadPThread.m3 had the same narrow startup race condition. I fixed it.Again I'm not seeing commit mails. On the larger matter: I think there is a solution, in that, if you look atThreadPThread.m3 and how it uses Posix condition variables,in that it does a fair amount of the work itself,you can term this "condition within condition". So I think maybe we can use the counting solutionand 1) merge the locks and/or 2) take the lock in Signal/Broadcast. But I have to look at it more. Specifically, in ThreadPThread.m3: Broadcast: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; next := t.nextWaiter; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Signal: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Becaise they signal while locked, we can implement this limitedform of condition variable, and then use the Modula-3 pthread logicusing that. Make some sense? Essentially we will have almost-posix condition variables,and exactly-posix mutexes, and then we can use just the pthread codeon top of that. Almost-posix condition variable is a posix condition variable, exceptit is guaranteed you will call pthread_cond_signal with the samelock held as when you call pthread_cond_wait. On a pthread system, the almost-posix condition variables and exactly-posixmutexes are just pass through to the underlying pthreads. On a win32 system, an almost-posix condition variable is similarto what we have now, except, because we know this lock is held,"XWait" can remove some of the leave/enter cycles and remove race conditions. Essentially we only need a "monitor", which is one lock and one condition variable. OR, we can use "directed notification" where each thread has an event, and threads are linked through a "wait block" i.e. the condition. This should work well too. It depends on us owing CreateThread, which we already do. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sat, 30 Jul 2016 21:18:08 +0000 I fixed the new_slots problem. I agree there are problems here. I think Java got away with a similar implementation because their interface is a bit different. i.e. the lock and condition are merged, and/or notification requires the caller to take the lock. We can't do this. We allow signal/broadcast w/o holding a lock. We'd have to resort to a global lock I guess. I suspect nobody else uses this solution. I believe we can solve it better and worse than others. Most libraries do not have their own "CreateThread" function.That is, most libraries let you call pthread_create or Win32 CreateThread,and the library will interoperate with any thread that comes its way.And most libraries don't seem to use thread locals in their solution.By "most libraries", I mean the pthreads on win32 package and Boost. Modula-3 isn't clearly this way. This isn't great, but it is the currentsituation. I believe it is fixable.That is, Modula-3 I believe requires using its library to create threads.This is not great for interoperation. You want to be able to be usedby code that isn't away of you, that just creates threads in the "normal" wayfor their platform. And then, either the current state, or a fix I have in mind, can be takenadvantage of to do "directed notify" w/o creating a kernel eventper wait or notify, like other solutions do. As to the fix, to not require threads go through our "CreateThread",I believe on Windows (which is all that is relevant here), we should havea DllMain that allocates thread locals. We can only easily have a DllMain if we are a .dll.Or we can use __declspec(thread).__declspec(thread) works if the .exe is statically linked to the exe or the dll,or on Vista and newer, but not in a .dll loaded by LoadLibrary prior to Vista.Perhaps that is good enough. If we require Vista then we can just drop a bunch of our code and usethe Win32 condition variables presumably.I remain curious how the condition variables work there, i.e. we can'timplement them ourselves, but it is possible they either require kernelsupport or are well beyond our abilities. - Jay > Date: Thu, 28 Jul 2016 15:00:45 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: A win32 thread test case > > I have attached a test case program designed to expose one of the bugs > I think I see in ThreadWin32.m3. It runs correctly on AMD64_LINUX. > I expect it to fail on Windows, but don't have a Windows machine I > can try it on. > > Anybody willing to try it? It's a self-contained Main module. Just > compile and run it. It will announce what it finds. > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 22:45:52 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 20:45:52 -0000 Subject: [M3devel] A win32 thread test case In-Reply-To: References: <579A646D.3020404@lcwb.coop>, , , , , Message-ID: I confirmed your test case fails with unmodified ThreadWin32.m3 and passes with my current work in progress. It is attached. I also removed the indirection on critical sections -- exposing the size to Modula3. I went with just an event as I indicated. So the ThreadPThread.m3 code, but with slight changes: - waits specify time to wait until time to wait until - event instead of pthread_cond - same maintenance of list of waiters Reasonable? - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 10:37:30 +0000 a fallacy here is that..broadcast isn't used, only signal, so the implementation is more general than needed.An even per thread should work. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 04:24:14 +0000 Something like attached?Still testing..might be crashing.A lot of the code and patterns come from ThreadPThread.m3. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 02:52:02 +0000 While I agree this is confusing, it is holding together for me and I have mostly coded it up. Repeating myself: In this scheme there are two types of condition variable. There are Modula-3 condition variables and posix-like/schmidt-like condition variables. The posix-like/smidt-like condition variables: Can be implemented directly via posix condition variables (if Windows pre-Vista had them). They are implemented as a slight modification of the Smidt counter method. The differences are that condition.lock is removed. Caller of cond_signal/cond_broadcast is responsible to hold the same lock as he holds when calling cond_wait. Some of the lock/unlock cycles are removed, and I hope therefore all the bugs are fixed. I still have to work through your examples. Modula-3 condition variables are then built on top of pthread-like/schmidt condition variables just like ThreadPThread.m3. This includes "alertable". The separate event previously in the Win32 implementation becomes a boolean plus cond_signal.XPause changes to call XWait, essentially. I was thinking we could reuse the Win32 thread alerting feature, but this alerts for other reasons so maybe isn't approprirate. If this works, we should be able to share more code between the pthread and win32 implementations.We just need a name for this slightly constrained pthread interface, i.e. where the condition variables assume the lock is held by the caller of signal/broadcast (we don't use broadcast). - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 00:47:24 +0000 ThreadPThread.m3 had the same narrow startup race condition. I fixed it.Again I'm not seeing commit mails. On the larger matter: I think there is a solution, in that, if you look atThreadPThread.m3 and how it uses Posix condition variables,in that it does a fair amount of the work itself,you can term this "condition within condition". So I think maybe we can use the counting solutionand 1) merge the locks and/or 2) take the lock in Signal/Broadcast. But I have to look at it more. Specifically, in ThreadPThread.m3: Broadcast: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; next := t.nextWaiter; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Signal: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Becaise they signal while locked, we can implement this limitedform of condition variable, and then use the Modula-3 pthread logicusing that. Make some sense? Essentially we will have almost-posix condition variables,and exactly-posix mutexes, and then we can use just the pthread codeon top of that. Almost-posix condition variable is a posix condition variable, exceptit is guaranteed you will call pthread_cond_signal with the samelock held as when you call pthread_cond_wait. On a pthread system, the almost-posix condition variables and exactly-posixmutexes are just pass through to the underlying pthreads. On a win32 system, an almost-posix condition variable is similarto what we have now, except, because we know this lock is held,"XWait" can remove some of the leave/enter cycles and remove race conditions. Essentially we only need a "monitor", which is one lock and one condition variable. OR, we can use "directed notification" where each thread has an event, and threads are linked through a "wait block" i.e. the condition. This should work well too. It depends on us owing CreateThread, which we already do. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sat, 30 Jul 2016 21:18:08 +0000 I fixed the new_slots problem. I agree there are problems here. I think Java got away with a similar implementation because their interface is a bit different. i.e. the lock and condition are merged, and/or notification requires the caller to take the lock. We can't do this. We allow signal/broadcast w/o holding a lock. We'd have to resort to a global lock I guess. I suspect nobody else uses this solution. I believe we can solve it better and worse than others. Most libraries do not have their own "CreateThread" function.That is, most libraries let you call pthread_create or Win32 CreateThread,and the library will interoperate with any thread that comes its way.And most libraries don't seem to use thread locals in their solution.By "most libraries", I mean the pthreads on win32 package and Boost. Modula-3 isn't clearly this way. This isn't great, but it is the currentsituation. I believe it is fixable.That is, Modula-3 I believe requires using its library to create threads.This is not great for interoperation. You want to be able to be usedby code that isn't away of you, that just creates threads in the "normal" wayfor their platform. And then, either the current state, or a fix I have in mind, can be takenadvantage of to do "directed notify" w/o creating a kernel eventper wait or notify, like other solutions do. As to the fix, to not require threads go through our "CreateThread",I believe on Windows (which is all that is relevant here), we should havea DllMain that allocates thread locals. We can only easily have a DllMain if we are a .dll.Or we can use __declspec(thread).__declspec(thread) works if the .exe is statically linked to the exe or the dll,or on Vista and newer, but not in a .dll loaded by LoadLibrary prior to Vista.Perhaps that is good enough. If we require Vista then we can just drop a bunch of our code and usethe Win32 condition variables presumably.I remain curious how the condition variables work there, i.e. we can'timplement them ourselves, but it is possible they either require kernelsupport or are well beyond our abilities. - Jay > Date: Thu, 28 Jul 2016 15:00:45 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: A win32 thread test case > > I have attached a test case program designed to expose one of the bugs > I think I see in ThreadWin32.m3. It runs correctly on AMD64_LINUX. > I expect it to fail on Windows, but don't have a Windows machine I > can try it on. > > Anybody willing to try it? It's a self-contained Main module. Just > compile and run it. It will announce what it finds. > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ThreadWin32.m3 Type: application/octet-stream Size: 42952 bytes Desc: not available URL: From jay.krell at cornell.edu Fri Jul 1 04:34:02 2016 From: jay.krell at cornell.edu (Jay K) Date: Fri, 1 Jul 2016 02:34:02 +0000 Subject: [M3devel] slight IR variation among targets? Message-ID: This is both a question and an explanation of hopefully an uncoming chnage, if I figure it out. Most platforms are almost the same. For example I386_FREEBSD and I386_NETBSD. You want their IR (and possibly code, but here just the IR) to generally be the same, along as long they aren't printing their actual target name. And they mostly are. Here is a different for example: jair:libm3 jay$ diff -du I386_NETBSD/NullRd.mc.txt I386_FREEBSD/NullRd.mc.txt? --- I386_NETBSD/NullRd.mc.txt 2016-06-30 19:13:11.000000000 -0700 +++ I386_FREEBSD/NullRd.mc.txt 2016-06-30 19:13:23.000000000 -0700 @@ -44,7 +44,6 @@ ? declare_procedure NullRd__Length 1 Int.32 0 0 F * p.7 ? declare_param rd 4 4 Addr 34129018 F F 50 v.10 ? declare_local _result 4 4 Int.32 425470580 F F 50 v.11 - reveal_opaque 34129018 -885236436 ? declare_opaque 483796623 -1651526519 ? declare_proctype -2116580915 2 -915982019 2 0 ? declare_formal n -1746782238 @@ -85,6 +84,7 @@ ? declare_field closed 416 8 509158269 ? declare_field seekable 424 8 509158269 ? declare_field intermittent 432 8 509158269 + reveal_opaque 34129018 -885236436 ? # Init ? -----LINE 22 ?----- ? begin_procedure p.5 One line of the IR is moved. The meaning is the same. What causes this? Do we output sometimes in hash order? And target affects that? Or when buffers fill up, and target length can affect that? I'd like to remove these differences. If it is a buffer length matter, I might just expand the buffers, as systems have far far more memory than when this was all written. If it is hash ordering matter -- one should avoid outputting data in hash order. ?- Jay From jay.krell at cornell.edu Fri Jul 1 08:48:57 2016 From: jay.krell at cornell.edu (Jay K) Date: Fri, 1 Jul 2016 06:48:57 +0000 Subject: [M3devel] lang.opt with -y twice? Message-ID: any idea why -y is here twice? Seems redundant. https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt?rev=1.2;content-type=text%2Fplain -y is shorthand for trace all, I know, Tony's advise years ago, very useful ?- Jay From rodney_bates at lcwb.coop Fri Jul 1 17:01:51 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Fri, 01 Jul 2016 10:01:51 -0500 Subject: [M3devel] source paths to generics? In-Reply-To: References: , , , , , , , <57746905.2080007@lcwb.coop> , <5775407E.6000401@lcwb.coop> Message-ID: <577685DF.2050404@lcwb.coop> On 06/30/2016 11:11 AM, Jay K wrote: > I put in a bunch of calls to RTIO.PutText("I'm here 1"); RTIO.Flush(), RTIO.PutText("I'm here 2"); RTIO.Flush(). > > When I saw those I knew I missed it somehow and looked again. > > I still don't see the strings in the output though. > > - Jay > > The missing call to M3Front.ParseImports is at cm3/src/Builder.m3:2007. It's in my previous grep output, plain as day. I must have just missed it in all the noise. But I'm still pretty sure the "=>" ("1=>"?) inserted by M3Header.Parse will never find its way into any output. > > ---------------------------------------- >> Date: Thu, 30 Jun 2016 10:53:34 -0500 >> From: rodney_bates at lcwb.coop >> To: m3devel at elegosoft.com >> Subject: Re: [M3devel] source paths to generics? >> >> >> >> On 06/29/2016 11:49 PM, Jay K wrote: >>> I guess neither of us looked outside of m3front. >>> The code runs. Not clear if the strings can get output. >>> Maybe from asserts or Compiler.ThisFile in a generic? >>> I"ll try some more. >>> >> >> I started looking in all of m3-sys, since the different compiler >> packages there are linked together into cm3. Later, I >> looked in the entire cm3 repo and didn't see anything. >> In any case, since it's part of the compiler, it really would be >> strange to call it from outside m3-sys. >> >> But if it is executing, we must have missed something. How did you >> find it executing? In a debugger? Can you do a backtrace? >> >>> jair:m3core jay$ grep -i parseimp ../../m3-sys/cm3/src/Builder.m3 >>> ids := M3Front.ParseImports (source, s.m3env); >>> >>> >>> - Jay >>> >>> >>> ---------------------------------------- >>>> Date: Wed, 29 Jun 2016 19:34:13 -0500 >>>> From: rodney_bates at lcwb.coop >>>> To: m3devel at elegosoft.com >>>> Subject: Re: [M3devel] source paths to generics? >>>> >>>> Hmm. I poked around a bit, and it looks like there is some cruft here, >>>> left over from something. >>>> >>>> The only call I can find on M3Header.Parse is M3Front.m3:37. This is >>>> in M3Front.ParseImports. I can find no calls on that at all. (There >>>> are other procedures by this name.) >>>> >>>> Also, it appears M3Header.ParseImports actually collects the exports, the imports, >>>> the generic actuals. >>>> >>>> As for the concocted (by M3Header) name of the form => , >>>> this looks to be used only to initialize the scanner's file number while scanning >>>> up through the formals of the generic, but that is not used. >>>> >>>> On 06/29/2016 11:34 AM, Jay K wrote: >>>>> There is similar code in Module.m3 and that is the code producing >>>>> the data "I don't like". >>>>> >>>>> M3Header isn't dead but I haven't seen it run yet. >>>>> >>>>> I changed them from "=>" to "1=>" and "2=>" and looked >>>>> where those occur in the output .s files under -keep. >>>>> >>>>> This is kinda something I wish were easier, like more strings >>>>> need to be longer for easier finding w/o ambiguity (the flip >>>>> side of my context arguments!) >>>>> >>>>> As things stand, I can't check that in. >>>>> I suppose maybe with a CG.comment but nevermind. >>>>> >>>>> - Jay >>>>> >>>>> >>>>> >>>>> ---------------------------------------- >>>>>> From: jay.krell at cornell.edu >>>>>> To: m3devel at elegosoft.com >>>>>> Subject: RE: [M3devel] source paths to generics? >>>>>> Date: Wed, 29 Jun 2016 15:22:14 +0000 >>>>>> >>>>>> mfront/src/misc/M3Header.m3 has this: >>>>>> >>>>>> >>>>>> (* build a synthetic file name & start reading *) >>>>>> filename := old_filename & " => " & filename; >>>>>> Scanner.Push (filename, s.generic, is_main := Scanner.in_main); >>>>>> >>>>>> >>>>>> and instantiated generics aren't what I thought. >>>>>> I thought the build system or compiler did a textual >>>>>> substitution of the generic parameters into the entire implementation, >>>>>> and saved that to the file system, >>>>>> and had the assert/debug paths point to it. >>>>>> >>>>>> So could step through what looks exactly like an .m3 file. >>>>>> >>>>>> But it doesn't. The instantiated file is a little stub, like: >>>>>> >>>>>> jair:libm3 jay$ more I386_DARWIN/TextAtomTbl.m3 >>>>>> (*generated by m3build*) >>>>>> >>>>>> MODULE TextAtomTbl = Table (Text, Atom) END TextAtomTbl. >>>>>> >>>>>> >>>>>> so I have one or two proposals. >>>>>> >>>>>> 1) old_filename above contains the target, it looks like: >>>>>> >>>>>> "../I386_DARWIN/TextAtomTbl.m3 => ../src/table/Table.mg" >>>>>> or >>>>>> "../I386_DARWIN/TextAtomTbl.m3 => ../src/table" >>>>>> >>>>>> These both occur. >>>>>> I'm not sure what the second is, seems like a mistake. >>>>>> >>>>>> I suggest the first string in both pairs be changed to be the leaf element, like: >>>>>> >>>>>> "TextAtomTbl.m3 => ../src/table/Table.mg" >>>>>> or >>>>>> "TextAtomTbl.m3 => ../src/table" >>>>>> >>>>>> and maybe the second string in both cases should be as it is in the second pair. >>>>>> >>>>>> 2) Possibly it should reall just be: >>>>>> ../src/table/Table.mg >>>>>> >>>>>> and developer can step through that, knowing that it is a somewhat abstracted >>>>>> form of what is running >>>>>> >>>>>> I'm willing to do this under like: >>>>>> >>>>>> CONST TemporaryForTargetConvergence = TRUE >>>>>> >>>>>> or >>>>>> VAR TemporaryForTargetConvergence: BOOLEAN; >>>>>> >>>>>> and a command line switch, but I suspect it isn't really useful to anyone asis, >>>>>> so might as well remove target unconditionally. >>>>>> >>>>>> >>>>>> ? >>>>>> >>>>>> - Jay >>>>>> >>>>>> >>>>>> >>>>>> ---------------------------------------- >>>>>>> From: jay.krell at cornell.edu >>>>>>> To: m3devel at elegosoft.com >>>>>>> Date: Wed, 29 Jun 2016 09:44:51 +0000 >>>>>>> Subject: [M3devel] source paths to generics? >>>>>>> >>>>>>> I haven't fully verified this, but it appears for example if I debug a generic, or fail an assert in a generic, >>>>>>> the source file I am told about is the instantiated i3/m3 file. >>>>>>> >>>>>>> This isn't particularly useful for the programmer or convenient for the compiler, right? >>>>>>> >>>>>>> The programmer would rather see the .ig/.mg files, and this is easy to provide in the compiler? >>>>>>> I guess it is slightly easier in the compiler, but easy to do better? >>>>>>> >>>>>>> I should look into make it so? >>>>>>> >>>>>>> My real agenda is I want to see: >>>>>>> ../src/foo.mg >>>>>>> >>>>>>> instead of >>>>>>> I386_DARWIN/foo.m3 >>>>>>> >>>>>>> I don't want the target variation, but other points seem true also, right? >>>>>>> >>>>>>> Right? The line numbers match between them, and the generic syntax is so close to "normal" that a programmer not used to it won't be confused, right? >>>>>>> >>>>>>> I'll try to poke around more. >>>>>>> >>>>>>> - Jay >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> M3devel mailing list >>>>>>> M3devel at elegosoft.com >>>>>>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>>>>> >>>>> >>>>> _______________________________________________ >>>>> M3devel mailing list >>>>> M3devel at elegosoft.com >>>>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>>>> >>>> >>>> -- >>>> Rodney Bates >>>> rodney.m.bates at acm.org >>>> _______________________________________________ >>>> M3devel mailing list >>>> M3devel at elegosoft.com >>>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>> >>> _______________________________________________ >>> M3devel mailing list >>> M3devel at elegosoft.com >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>> >> >> -- >> Rodney Bates >> rodney.m.bates at acm.org >> _______________________________________________ >> M3devel mailing list >> M3devel at elegosoft.com >> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > -- Rodney Bates rodney.m.bates at acm.org From wagner at elegosoft.com Fri Jul 1 17:17:10 2016 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 1 Jul 2016 17:17:10 +0200 Subject: [M3devel] source paths to generics? In-Reply-To: <577685DF.2050404@lcwb.coop> References: <57746905.2080007@lcwb.coop> <5775407E.6000401@lcwb.coop> <577685DF.2050404@lcwb.coop> Message-ID: <20160701171710.add85dcbf03e0e3f1f24c451@elegosoft.com> On Fri, 01 Jul 2016 10:01:51 -0500 "Rodney M. Bates" wrote: > On 06/30/2016 11:11 AM, Jay K wrote: > > I put in a bunch of calls to RTIO.PutText("I'm here 1"); RTIO.Flush(), RTIO.PutText("I'm here 2"); RTIO.Flush(). > > > > When I saw those I knew I missed it somehow and looked again. > > > > I still don't see the strings in the output though. > > > > - Jay > > > > > > The missing call to M3Front.ParseImports is at cm3/src/Builder.m3:2007. > It's in my previous grep output, plain as day. > I must have just missed it in all the noise. > > But I'm still pretty sure the "=>" ("1=>"?) inserted by M3Header.Parse will > never find its way into any output. I'm sure there's lots of old and obsolete code in the packages, as CM3 was built upon the DEC SRC compiler sources. The principal programmer of the compiler and Reactor is easily found via LinkedIn: https://www.linkedin.com/in/billkalsow I'm not sure if he is willing to answer questions about his work, but if you run into a real problem, it might be worthwhile to try it. Olaf -- Olaf Wagner -- elego Software Solutions GmbH -- http://www.elegosoft.com Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From rodney_bates at lcwb.coop Fri Jul 1 20:24:48 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Fri, 01 Jul 2016 13:24:48 -0500 Subject: [M3devel] source paths to generics? In-Reply-To: <20160701171710.add85dcbf03e0e3f1f24c451@elegosoft.com> References: <57746905.2080007@lcwb.coop> <5775407E.6000401@lcwb.coop> <577685DF.2050404@lcwb.coop> <20160701171710.add85dcbf03e0e3f1f24c451@elegosoft.com> Message-ID: <5776B570.1050504@lcwb.coop> On 07/01/2016 10:17 AM, Olaf Wagner wrote: > On Fri, 01 Jul 2016 10:01:51 -0500 > "Rodney M. Bates" wrote: > >> On 06/30/2016 11:11 AM, Jay K wrote: >>> I put in a bunch of calls to RTIO.PutText("I'm here 1"); RTIO.Flush(), RTIO.PutText("I'm here 2"); RTIO.Flush(). >>> >>> When I saw those I knew I missed it somehow and looked again. >>> >>> I still don't see the strings in the output though. >>> >>> - Jay >>> >>> >> >> The missing call to M3Front.ParseImports is at cm3/src/Builder.m3:2007. >> It's in my previous grep output, plain as day. >> I must have just missed it in all the noise. >> >> But I'm still pretty sure the "=>" ("1=>"?) inserted by M3Header.Parse will >> never find its way into any output. > > I'm sure there's lots of old and obsolete code in the packages, as CM3 > was built upon the DEC SRC compiler sources. > I didn't make this clear, but, with the newly discovered call, my disparaging remarks about M3Header.Parse, etc. are now refuted. > The principal programmer of the compiler and Reactor is easily found > via LinkedIn: https://www.linkedin.com/in/billkalsow > > I'm not sure if he is willing to answer questions about his work, but > if you run into a real problem, it might be worthwhile to try it. > > Olaf > -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Fri Jul 1 20:43:20 2016 From: jay.krell at cornell.edu (Jay K) Date: Fri, 1 Jul 2016 18:43:20 +0000 Subject: [M3devel] source paths to generics? In-Reply-To: <5776B570.1050504@lcwb.coop> References: <57746905.2080007@lcwb.coop> <5775407E.6000401@lcwb.coop> <577685DF.2050404@lcwb.coop>, <20160701171710.add85dcbf03e0e3f1f24c451@elegosoft.com>, <5776B570.1050504@lcwb.coop> Message-ID: >> But I'm still pretty sure the "=>" ("1=>"?) inserted by M3Header.Parse will>> never find its way into any output. agreed.Anyway, I'm almost done with my minor change here. I'm leaning toward: cm3 -trim-target-variation will set Target.TrimTargetVariation will also a quake variable CM3_TRIM_TARGET_VARIATION maybe quake function backend() will use quake variable to pass it on to cm3cg probably Unless maybe builder code knows it is calling cm3cg and will it in options there i.e. don't intend to push this through llvm backends at this time. Used by M3C to turn off line directives entirely (vs. just commenting them out as it is willing to do) Used by M3C to turn off a comment as what is TARGET cm3cg -ftrim-target-variation used by dbxout.c and dwarf2out.c to omit current working directory It is a funny name and a funny behavior and of limited use. The exercise is somewhat useful to see how to plumb options through. Much target variation occurs, just that slightly unnecessary variation will not, and then targets that are nearly the same, will be closer to the same. In particular, it likely that there are approximately one-two ABI per architecture on Posix hosts, therefore the Posix targets are all the same -- NetBSD, OpenBSD, Solaris, Darwin, Linux, FreeBSD. Darwin will tend to vary because it assumes newer processors so favors SSE over x87. But still for AMD64, probably the same. Windows tends to have different ABIs so the IR is still the same, but the assembly is not. - Jay > Date: Fri, 1 Jul 2016 13:24:48 -0500 > From: rodney_bates at lcwb.coop > To: wagner at elegosoft.com; jay.krell at cornell.edu > Subject: Re: [M3devel] source paths to generics? > CC: m3devel at elegosoft.com > > > > On 07/01/2016 10:17 AM, Olaf Wagner wrote: > > On Fri, 01 Jul 2016 10:01:51 -0500 > > "Rodney M. Bates" wrote: > > > >> On 06/30/2016 11:11 AM, Jay K wrote: > >>> I put in a bunch of calls to RTIO.PutText("I'm here 1"); RTIO.Flush(), RTIO.PutText("I'm here 2"); RTIO.Flush(). > >>> > >>> When I saw those I knew I missed it somehow and looked again. > >>> > >>> I still don't see the strings in the output though. > >>> > >>> - Jay > >>> > >>> > >> > >> The missing call to M3Front.ParseImports is at cm3/src/Builder.m3:2007. > >> It's in my previous grep output, plain as day. > >> I must have just missed it in all the noise. > >> > >> But I'm still pretty sure the "=>" ("1=>"?) inserted by M3Header.Parse will > >> never find its way into any output. > > > > I'm sure there's lots of old and obsolete code in the packages, as CM3 > > was built upon the DEC SRC compiler sources. > > > > I didn't make this clear, but, with the newly discovered call, my > disparaging remarks about M3Header.Parse, etc. are now refuted. > > > The principal programmer of the compiler and Reactor is easily found > > via LinkedIn: https://www.linkedin.com/in/billkalsow > > > > I'm not sure if he is willing to answer questions about his work, but > > if you run into a real problem, it might be worthwhile to try it. > > > > Olaf > > > > -- > Rodney Bates > rodney.m.bates at acm.org > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Jul 4 10:45:23 2016 From: jay.krell at cornell.edu (Jay K) Date: Mon, 4 Jul 2016 08:45:23 +0000 Subject: [M3devel] defaulting to parallel backend? Message-ID: I suggest we make parallel backend the default. You can turn it off with -pb 1. Turn it "up" with -pb And the default is a thread per processor. This will be kernel specific code, but we can get far with ?1) Win32 GetSystemInfo, if applicable (e.g. C backend) ?2)?sysconf(_SC_NPROCESSORS_ONLN) and I'll research the various OSes (Darwin, Linux, *BSD, Solaris, and possibly AIX, HP-UX, Irix, Mingw, Cygwin). ok? I could really use the speed boost and multi-processor machines are overwhelmingly common now (though easily avoided with VMs). ?- Jay From jay.krell at cornell.edu Mon Jul 4 10:53:44 2016 From: jay.krell at cornell.edu (Jay K) Date: Mon, 4 Jul 2016 08:53:44 +0000 Subject: [M3devel] cm3 -reduce-target-variation / cm3cg -freduce-target-variation? Message-ID: I guess I should learn to use branches and/or pull requests. This adds cm3 -reduce-target-variation and cm3cg -freduce-target-variation. This meant for temporary development purposes. No user would use them. The point is to make e..g I386_FREEBSD and I386_NETBSD demonstrably identical, or at least closer to it. Ok to commit? Maybe name the two switches the same? Or something else? Maybe be less aggressive? ?i.e. even ../src/foo.m3 is getting changed to foo.m3. ?The real target is ../I386_DARWIN/foo.m3 => foo.m3? ? ? diff --git a/m3-sys/cm3/src/Main.m3 b/m3-sys/cm3/src/Main.m3 index 5d753e6..61b3f18 100644 --- a/m3-sys/cm3/src/Main.m3 +++ b/m3-sys/cm3/src/Main.m3 @@ -5,7 +5,7 @@ MODULE Main; ? ?IMPORT M3Timers, Pathname, Process, Quake; ?IMPORT RTCollector, RTParams, RTutils, Thread, Wr; -IMPORT TextTextTbl; +IMPORT TextTextTbl, Target; ? ?IMPORT Builder, Dirs, M3Build, M3Options, Makefile, Msg, Utils, WebFile; ?IMPORT MxConfig(*, M3Config, CMKey, CMCurrent *); @@ -18,6 +18,8 @@ VAR ? ?build_dir : TEXT ? ? ? ? ?:= NIL; ? ?mach ? ? ?: Quake.Machine := NIL; ? +CONST BoolToText = ARRAY BOOLEAN OF TEXT{"FALSE", "TRUE"}; + ?PROCEDURE DefineIfNotDefined (qmachine: Quake.Machine; ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?symbol, value: TEXT) RAISES {Quake.Error} = ?BEGIN @@ -73,6 +75,7 @@ VAR defs: TextTextTbl.T; ? ? ? ? ?(* DefineIfNotDefined (mach, "THREAD_LIBRARY", Version.ThreadLibrary); *) ? ? ? ? ?(* DefineIfNotDefined (mach, "WINDOW_LIBRARY", Version.WindowLibrary); *) ? ? ? ? ?DefineIfNotDefined (mach, "WORD_SIZE", MxConfig.HOST_WORD_SIZE); + ? ? ? ?DefineIfNotDefined (mach, "REDUCE_TARGET_VARIATION", BoolToText[Target.ReduceTargetVariation]); ? ? ? ? ? ?(* Even if the config file overrides the defaults, such as to do ? ? ? ? ? ? a cross build, the host characteristics are still available. *) diff --git a/m3-sys/cm3/src/Makefile.m3 b/m3-sys/cm3/src/Makefile.m3 index 64e169e..8b12320 100644 --- a/m3-sys/cm3/src/Makefile.m3 +++ b/m3-sys/cm3/src/Makefile.m3 @@ -5,7 +5,7 @@ MODULE Makefile; ? ?IMPORT FS, M3File, M3Timers, OSError, Params, Process, Text, Thread, Wr; ?IMPORT Arg, M3Build, M3Options, M3Path, Msg, Utils, TextSeq, TextTextTbl; -IMPORT MxConfig, Dirs, Version; +IMPORT MxConfig, Dirs, Version, Target; ? ?TYPE ? ?NK = M3Path.Kind; @@ -267,6 +267,9 @@ PROCEDURE ConvertOption (VAR s: State; ?arg: TEXT; ?arg_len: INTEGER) ? ? ?| 'r' => IF Text.Equal(arg, "-realclean") THEN ? ? ? ? ? ? ? ? ok := TRUE; ?(* mode set during the pre-scan *) ? ? ? ? ? ? ? ? s.found_work := TRUE; + ? ? ? ? ? ? ELSIF Text.Equal(arg, "-reduce-target-variation") THEN + ? ? ? ? ? ? ? ok := TRUE; + ? ? ? ? ? ? ? Target.ReduceTargetVariation := TRUE; ? ? ? ? ? ? ? END; ? ? ? ? ? ? ?? ? ? ?| 's' => IF Text.Equal (arg, "-silent") THEN @@ -707,6 +710,9 @@ CONST ? ? ?" ?-group-writable \"", ? ? ?" ?-pb ? ? ? ?allow parallelism in running back-end (experimental)", ? ? ?" ?-no-m3ship-resolution use quake variables in .M3SHIP (experimental)", + ? ?" ?-reduce-target-variation omit target in some minor places such as", + ? ?" ? ? ? ? ? ? ? ? ?current working directory in debug information", + ? ?" ? ? ? ? ? ? ? ? ?for internal development purposes (showing target equivalence)", ? ? ?"", ? ? ?"environment variables:", ? ? ?" ?M3CONFIG ? ? ? platform dependent configuration file to use (cm3.cfg)", diff --git a/m3-sys/cminstall/src/config-no-install/cm3cfg.common b/m3-sys/cminstall/src/config-no-install/cm3cfg.common index 7334252..9806c3c 100644 --- a/m3-sys/cminstall/src/config-no-install/cm3cfg.common +++ b/m3-sys/cminstall/src/config-no-install/cm3cfg.common @@ -460,6 +460,12 @@ proc GetM3Back() is ? ?end end ? ? ?m3back = "@" & m3back & "cm3cg " & GetM3BackFlags() + ? + ?if defined ("REDUCE_TARGET_VARIATION") + ? ?if REDUCE_TARGET_VARIATION + ? ? ?m3back = m3back & " -freduce-target-variation" + ? ?end + ?end ? ?return m3back ?end ? diff --git a/m3-sys/m3back/src/M3C.m3 b/m3-sys/m3back/src/M3C.m3 index f6740c7..22c6ade 100644 --- a/m3-sys/m3back/src/M3C.m3 +++ b/m3-sys/m3back/src/M3C.m3 @@ -29,7 +29,7 @@ VAR CaseDefaultAssertFalse := FALSE; ? ?(* Taken together, these help debugging, as you get more lines in the ? ? C and the error messages reference C line numbers *) - ?CONST output_line_directives = TRUE; + ?VAR ?output_line_directives := TRUE; ? ?CONST output_extra_newlines = FALSE; ? ?CONST inline_extract = FALSE; ? @@ -2197,7 +2197,9 @@ BEGIN ? ? ?END; ? ? ? ?IF (*self.suppress_line_directive < 1 AND*) text_last_char = '\n' THEN + ? ? ? ?IF NOT Target.ReduceTargetVariation THEN ? ? ? ? ? ?Wr.PutText(self.c, self.line_directive); + ? ? ? ?END; ? ? ? ? ?self.width := 0; ? ? ? ? ?self.last_char_was_newline := TRUE; ? ? ? ? ?RETURN; @@ -2205,7 +2207,9 @@ BEGIN ? ? ? ?IF Text.FindChar(text, '\n') # -1 THEN ? ? ? ? ?self.width := 0; (* roughly *) + ? ? ? ?IF NOT Target.ReduceTargetVariation THEN ? ? ? ? ? ?Wr.PutText(self.c, self.nl_line_directive); + ? ? ? ?END; ? ? ? ? ?self.last_char_was_newline := TRUE; ? ? ? ? ?RETURN; ? ? ?END; @@ -2217,11 +2221,13 @@ BEGIN ? ? ?END; ? ? ? ?self.width := 0; + ? ?IF NOT Target.ReduceTargetVariation THEN ? ? ? ?IF self.last_char_was_newline THEN ? ? ? ? ?Wr.PutText(self.c, self.line_directive); ? ? ? ?ELSE ? ? ? ? ?Wr.PutText(self.c, self.nl_line_directive); ? ? ? ?END; + ? ?END; ? ? ?self.last_char_was_newline := TRUE; ?END print; ? @@ -2339,9 +2345,10 @@ END set_error_handler; ?PROCEDURE Prefix_Print(self: T; multipass: Multipass_t) = ?BEGIN ? ? ?self.comment("begin unit"); + ? ?output_line_directives := output_line_directives AND NOT Target.ReduceTargetVariation; + ? ?IF NOT Target.ReduceTargetVariation THEN ? ? ? ?self.comment("M3_TARGET = ", Target.System_name); - ? ?(* This is an unnecessary target-specific output. *) - ? ?(* self.comment("M3_TARGET = ", Target.System_name); *) + ? ?END; ? ? ?self.comment("M3_WORDSIZE = ", IntToDec(Target.Word.size)); ? ? ?self.static_link_id := M3ID.Add("_static_link"); ? ? ?self.alloca_id := M3ID.Add("alloca"); diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c index dc52576..33f8844 100644 --- a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c +++ b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c @@ -1065,6 +1065,9 @@ dbxout_init (const char *input_file_name) ? ? ? labels. ?*/ ? ?ASM_GENERATE_INTERNAL_LABEL (ltext_label_name, "Ltext", 0); ? + ?/* Limit paths in debug output, to limit target variation. */ + ?if (!reduce_target_variation) + ?{ ? ? ?/* Put the current working directory in an N_SO symbol. ?*/ ? ? ?if (use_gnu_debug_info_extensions && !NO_DBX_MAIN_SOURCE_DIRECTORY) ? ? ?{ @@ -1087,6 +1090,7 @@ dbxout_init (const char *input_file_name) ? ? ? ?used_ltext_label_name = true; ?#endif /* no DBX_OUTPUT_MAIN_SOURCE_DIRECTORY */ ? ? ?} + ?} ? ? ?mapped_name = remap_debug_filename (input_file_name); ?#ifdef DBX_OUTPUT_MAIN_SOURCE_FILENAME diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c index 0da1021..0a48a65 100644 --- a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c +++ b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c @@ -15450,19 +15450,20 @@ add_gnat_descriptive_type_attribute (dw_die_ref die, tree type, ?static void ?add_comp_dir_attribute (dw_die_ref die) ?{ + ?/* Limit paths in debug output, to limit target variation. */ + ?if (reduce_target_variation) + ? ?return; + ? ?const char *wd = get_src_pwd (); - ?char *wd1; ? ? ?if (wd == NULL) ? ? ?return; ? ? ?if (DWARF2_DIR_SHOULD_END_WITH_SEPARATOR) ? ? ?{ - ? ? ?int wdlen; - - ? ? ?wdlen = strlen (wd); - ? ? ?wd1 = (char *) ggc_alloc_atomic (wdlen + 2); - ? ? ?strcpy (wd1, wd); + ? ? ?int const wdlen = (int)strlen (wd); + ? ? ?char * const wd1 = (char *) ggc_alloc_atomic (wdlen + 2); + ? ? ?memcpy (wd1, wd, wdlen); ? ? ? ?wd1 [wdlen] = DIR_SEPARATOR; ? ? ? ?wd1 [wdlen + 1] = 0; ? ? ? ?wd = wd1; diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c index 63e4b92..d468632 100644 --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c @@ -217,11 +217,8 @@ const char * ?get_src_pwd (void) ?{ ? ?if (! src_pwd) - ? ?{ - ? ? ?src_pwd = getpwd (); - ? ? ?if (!src_pwd) + ? ?if (reduce_target_variation || !(src_pwd = getpwd ())) ? ? ? ?src_pwd = "."; - ? ?} ? ? ? return src_pwd; ?} diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h index 588cfdb..f4f7cc7 100644 --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h @@ -80,4 +80,6 @@ extern bool set_src_pwd ? ? ? (const char *); ?extern HOST_WIDE_INT get_random_seed (bool); ?extern const char *set_random_seed (const char *); ? +extern bool reduce_target_variation; + ?#endif /* ! GCC_TOPLEV_H */ diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt index 3bd0469..edfca96 100644 --- a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt +++ b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt @@ -28,9 +28,6 @@ m3cg ?Language ?M3CG ? -y -m3cg M3CG - ?fopcodes-trace ?m3cg M3CG ?Trace opcodes @@ -59,10 +56,17 @@ ftypes-trace ?m3cg M3CG ?Trace types ? +freduce-target-variation +m3cg M3CG +Reduce target variation somewhat, such as by omitting current working +directory from debug info. Many necessary target variations remain. + ?v ?m3cg M3CG +print version ? ?y ?m3cg M3CG +Trace opcodes ? ?; This comment is to ensure we retain the blank line above. diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c index 03417ed..0cfa46a 100644 --- a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c +++ b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c @@ -246,6 +246,7 @@ build_case_label (tree low_value, tree high_value, tree label_decl) ?/*======================================================= OPTION HANDLING ===*/ ? ?static int option_trace_all; +bool reduce_target_variation; ? ?/*===========================================================================*/ ? @@ -6364,6 +6365,10 @@ m3_handle_option (size_t code, PCSTR /*arg*/, int /*value*/) ? ? ?case OPT_ftypes_trace: ? ? ? ?option_trace_all += 1; ? ? ? ?break; + ? ? ? + ? ?case OPT_freduce_target_variation: + ? ? ?reduce_target_variation = true; + ? ? ?break; ? ? ?} ? ? ?return 1; diff --git a/m3-sys/m3front/src/misc/Coverage.m3 b/m3-sys/m3front/src/misc/Coverage.m3 index c04c902..73fff21 100644 --- a/m3-sys/m3front/src/misc/Coverage.m3 +++ b/m3-sys/m3front/src/misc/Coverage.m3 @@ -77,8 +77,9 @@ PROCEDURE NoteProcedure (v: Value.T) = ?PROCEDURE GenerateTables () = ? ?VAR ? ? ?nLines ? ?:= MAX (0, maxLine - minLine) + 1; + ? ?fname ? ? := Host.FileTail (Host.filename); ? ? ?l_header ?:= TLen (Header); - ? ?l_fname ? := TLen (Host.filename); + ? ?l_fname ? := TLen (fname); ? ? ?l_trailer := TLen (Trailer); ? ? ?size ? ?: INTEGER; ? ? ?p ? ? ? : ProcHead; @@ -124,10 +125,10 @@ PROCEDURE GenerateTables () = ? ? ?(* CG.Init_int (size, Target.Integer.size, TInt.Zero, FALSE); *) ? ? ?INC (size, Target.Integer.size); ? ? ? ? ? ? (*timestamp*) ? - ? ?CG.Init_intt (size, Target.Integer.size, Text.Length (Host.filename), FALSE); + ? ?CG.Init_intt (size, Target.Integer.size, Text.Length (fname), FALSE); ? ? ?INC (size, Target.Integer.size); ? ? ? ? ? ? (*fileLen*) ? - ? ?CG.Init_chars (size, Host.filename, FALSE); + ? ?CG.Init_chars (size, fname, FALSE); ? ? ?INC (size, l_fname * Target.Char.size); ? ? ?(*file*) ? ? ? ?CG.Init_intt (size, Target.Integer.size, minLine, FALSE); diff --git a/m3-sys/m3front/src/misc/Host.i3 b/m3-sys/m3front/src/misc/Host.i3 index c71489f..af6a89a 100644 --- a/m3-sys/m3front/src/misc/Host.i3 +++ b/m3-sys/m3front/src/misc/Host.i3 @@ -75,4 +75,7 @@ PROCEDURE OpenUnit (name: M3ID.T; interface, generic: BOOLEAN; ? ?PROCEDURE CloseFile (rd: File.T); ? +PROCEDURE FileTail (path: TEXT): TEXT; + (* returns the 'tail' of 'path' -- after any slashes or even spaces *) + ?END Host. diff --git a/m3-sys/m3front/src/misc/Host.m3 b/m3-sys/m3front/src/misc/Host.m3 index 962d3c6..79042e3 100644 --- a/m3-sys/m3front/src/misc/Host.m3 +++ b/m3-sys/m3front/src/misc/Host.m3 @@ -9,7 +9,7 @@ ? ?MODULE Host; ? -IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler; +IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler, Target; ? ?PROCEDURE Initialize (READONLY options: ARRAY OF TEXT): BOOLEAN = ? ?BEGIN @@ -192,5 +192,24 @@ PROCEDURE CloseFile (rd: File.T) = ? ? ?END; ? ?END CloseFile; ? +PROCEDURE FileTail (path: TEXT): TEXT = + ?VAR c: CHAR; + ?BEGIN + ? ?IF NOT Target.ReduceTargetVariation THEN RETURN path; END; + + ? ?IF (path = NIL) THEN RETURN NIL END; + + ? ?(* search for the last slash or blank in the string *) + ? ?FOR x := Text.Length (path) - 1 TO 0 BY -1 DO + ? ? ?c := Text.GetChar (path, x); + ? ? ?IF (c = '/') OR (c = ' ') OR (c = '\\') THEN + ? ? ? ?RETURN Text.Sub (path, x+1); + ? ? ?END; + ? ?END; + + ? ?(* no slashes *) + ? ?RETURN path; + ?END FileTail; + ?BEGIN ?END Host. diff --git a/m3-sys/m3front/src/misc/M3Header.m3 b/m3-sys/m3front/src/misc/M3Header.m3 index 1e4decf..877d77c 100644 --- a/m3-sys/m3front/src/misc/M3Header.m3 +++ b/m3-sys/m3front/src/misc/M3Header.m3 @@ -104,7 +104,7 @@ PROCEDURE PushGeneric (VAR s: State) = ? ? ?IF (s.generic = NIL) THEN s.failed := TRUE; ?RETURN; END; ? ? ? ?(* build a synthetic file name & start reading *) - ? ?filename := old_filename & " => " & filename; + ? ?filename := Host.FileTail(old_filename) & " => " & filename; ? ? ?Scanner.Push (filename, s.generic, is_main := Scanner.in_main); ? ? ? ?(* make sure we got what we wanted *) diff --git a/m3-sys/m3front/src/misc/Scanner.m3 b/m3-sys/m3front/src/misc/Scanner.m3 index 7470374..e1dc024 100644 --- a/m3-sys/m3front/src/misc/Scanner.m3 +++ b/m3-sys/m3front/src/misc/Scanner.m3 @@ -228,13 +228,16 @@ PROCEDURE Here (VAR file: TEXT; ?VAR line: INTEGER) = ? ?BEGIN ? ? ?file := files [offset DIV MaxLines]; ? ? ?line := offset MOD MaxLines; + ? ?IF Target.ReduceTargetVariation THEN + ? ? ?file := Host.FileTail(file); + ? ?END; ? ?END Here; ? ?PROCEDURE LocalHere (VAR file: TEXT; ?VAR line: INTEGER) = ? ?VAR fnum := offset DIV MaxLines; ? ?BEGIN ? ? ?IF (local_files[fnum] = NIL) THEN - ? ? ?local_files[fnum] := files[fnum]; + ? ? ?local_files[fnum] := Host.FileTail(files[fnum]); ? ? ?END; ? ? ?file := local_files [fnum]; ? ? ?line := offset MOD MaxLines; diff --git a/m3-sys/m3front/src/values/Module.m3 b/m3-sys/m3front/src/values/Module.m3 index a085eab..576c857 100644 --- a/m3-sys/m3front/src/values/Module.m3 +++ b/m3-sys/m3front/src/values/Module.m3 @@ -421,7 +421,7 @@ PROCEDURE PushGeneric (t: T; ?VAR rd: File.T): M3ID.T = ? ? ?END; ? ? ? ?(* build a synthetic file name & start reading *) - ? ?filename := old_filename & " => " & filename; + ? ?filename := Host.FileTail(old_filename) & " => " & filename; ? ? ?Scanner.Push (filename, rd, is_main := Scanner.in_main); ? ? ?t.genericFile := filename; ? diff --git a/m3-sys/m3middle/src/Target.i3 b/m3-sys/m3middle/src/Target.i3 index fe198d2..cd7acee 100644 --- a/m3-sys/m3middle/src/Target.i3 +++ b/m3-sys/m3middle/src/Target.i3 @@ -529,4 +529,8 @@ VAR (*CONST*) ? ? ? test for nested procedures passed as parameters must be more ? ? ? elaborate (e.g. HPPA). *) ? + (* This removes some unnecessary target variation in the output, + ?* such as current working directory in debug output. *) + ReduceTargetVariation: BOOLEAN; + ?END Target. diff --git a/scripts/python/pylib.py b/scripts/python/pylib.py index 73e622f..487ad17 100755 --- a/scripts/python/pylib.py +++ b/scripts/python/pylib.py @@ -388,7 +388,7 @@ def _GetAllTargets(): ? ?_CBackend = "c" in sys.argv or "C" in sys.argv ?_BuildDirC = ["", "c"][_CBackend] -_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why"] +_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why", "reduce-target-variation", "reducetargetvariation"] ?_SkipGccFlags = ["nogcc", "skipgcc", "omitgcc"] ?_PossiblePylibFlags = ["noclean", "nocleangcc", "c", "C"] + _SkipGccFlags + _PossibleCm3Flags ?Thank you, ?- Jay From rodney_bates at lcwb.coop Mon Jul 4 16:55:38 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 04 Jul 2016 09:55:38 -0500 Subject: [M3devel] defaulting to parallel backend? In-Reply-To: References: Message-ID: <577A78EA.5000600@lcwb.coop> I prefer to use one fewer than the number of processors, so I can web browse, read email, review code, etc. while waiting for a compile. The compiler can saturate them all, making any other stuff very slow. On 07/04/2016 03:45 AM, Jay K wrote: > I suggest we make parallel backend the default. > > You can turn it off with -pb 1. > > Turn it "up" with -pb > > And the default is a thread per processor. > This will be kernel specific code, but we can get far with > > 1) Win32 GetSystemInfo, if applicable (e.g. C backend) > 2) sysconf(_SC_NPROCESSORS_ONLN) > > > and I'll research the various OSes (Darwin, Linux, *BSD, Solaris, and possibly AIX, HP-UX, Irix, Mingw, Cygwin). > ok? > > I could really use the speed boost and multi-processor machines are overwhelmingly common now (though easily avoided with VMs). > > > - Jay > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Mon Jul 4 16:58:32 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 04 Jul 2016 09:58:32 -0500 Subject: [M3devel] cm3 -reduce-target-variation / cm3cg -freduce-target-variation? In-Reply-To: References: Message-ID: <577A7998.1030907@lcwb.coop> On 07/04/2016 03:53 AM, Jay K wrote: > I guess I should learn to use branches and/or pull requests. > > > > This adds cm3 -reduce-target-variation and cm3cg -freduce-target-variation. > > > This meant for temporary development purposes. > No user would use them. > The point is to make e..g I386_FREEBSD and I386_NETBSD demonstrably identical, or at least closer to it. > > > Ok to commit? > If nothing happens unless these switches are explicitly supplied, it should have no effect on anybody else, so OK. > > Maybe name the two switches the same? Or something else? > > > Maybe be less aggressive? > i.e. even ../src/foo.m3 is getting changed to foo.m3. > The real target is ../I386_DARWIN/foo.m3 => foo.m3 > > > > diff --git a/m3-sys/cm3/src/Main.m3 b/m3-sys/cm3/src/Main.m3 > index 5d753e6..61b3f18 100644 > --- a/m3-sys/cm3/src/Main.m3 > +++ b/m3-sys/cm3/src/Main.m3 > @@ -5,7 +5,7 @@ MODULE Main; > > IMPORT M3Timers, Pathname, Process, Quake; > IMPORT RTCollector, RTParams, RTutils, Thread, Wr; > -IMPORT TextTextTbl; > +IMPORT TextTextTbl, Target; > > IMPORT Builder, Dirs, M3Build, M3Options, Makefile, Msg, Utils, WebFile; > IMPORT MxConfig(*, M3Config, CMKey, CMCurrent *); > @@ -18,6 +18,8 @@ VAR > build_dir : TEXT := NIL; > mach : Quake.Machine := NIL; > > +CONST BoolToText = ARRAY BOOLEAN OF TEXT{"FALSE", "TRUE"}; > + > PROCEDURE DefineIfNotDefined (qmachine: Quake.Machine; > symbol, value: TEXT) RAISES {Quake.Error} = > BEGIN > @@ -73,6 +75,7 @@ VAR defs: TextTextTbl.T; > (* DefineIfNotDefined (mach, "THREAD_LIBRARY", Version.ThreadLibrary); *) > (* DefineIfNotDefined (mach, "WINDOW_LIBRARY", Version.WindowLibrary); *) > DefineIfNotDefined (mach, "WORD_SIZE", MxConfig.HOST_WORD_SIZE); > + DefineIfNotDefined (mach, "REDUCE_TARGET_VARIATION", BoolToText[Target.ReduceTargetVariation]); > > (* Even if the config file overrides the defaults, such as to do > a cross build, the host characteristics are still available. *) > diff --git a/m3-sys/cm3/src/Makefile.m3 b/m3-sys/cm3/src/Makefile.m3 > index 64e169e..8b12320 100644 > --- a/m3-sys/cm3/src/Makefile.m3 > +++ b/m3-sys/cm3/src/Makefile.m3 > @@ -5,7 +5,7 @@ MODULE Makefile; > > IMPORT FS, M3File, M3Timers, OSError, Params, Process, Text, Thread, Wr; > IMPORT Arg, M3Build, M3Options, M3Path, Msg, Utils, TextSeq, TextTextTbl; > -IMPORT MxConfig, Dirs, Version; > +IMPORT MxConfig, Dirs, Version, Target; > > TYPE > NK = M3Path.Kind; > @@ -267,6 +267,9 @@ PROCEDURE ConvertOption (VAR s: State; arg: TEXT; arg_len: INTEGER) > | 'r' => IF Text.Equal(arg, "-realclean") THEN > ok := TRUE; (* mode set during the pre-scan *) > s.found_work := TRUE; > + ELSIF Text.Equal(arg, "-reduce-target-variation") THEN > + ok := TRUE; > + Target.ReduceTargetVariation := TRUE; > END; > > | 's' => IF Text.Equal (arg, "-silent") THEN > @@ -707,6 +710,9 @@ CONST > " -group-writable \"", > " -pb allow parallelism in running back-end (experimental)", > " -no-m3ship-resolution use quake variables in .M3SHIP (experimental)", > + " -reduce-target-variation omit target in some minor places such as", > + " current working directory in debug information", > + " for internal development purposes (showing target equivalence)", > "", > "environment variables:", > " M3CONFIG platform dependent configuration file to use (cm3.cfg)", > diff --git a/m3-sys/cminstall/src/config-no-install/cm3cfg.common b/m3-sys/cminstall/src/config-no-install/cm3cfg.common > index 7334252..9806c3c 100644 > --- a/m3-sys/cminstall/src/config-no-install/cm3cfg.common > +++ b/m3-sys/cminstall/src/config-no-install/cm3cfg.common > @@ -460,6 +460,12 @@ proc GetM3Back() is > end end > > m3back = "@" & m3back & "cm3cg " & GetM3BackFlags() > + > + if defined ("REDUCE_TARGET_VARIATION") > + if REDUCE_TARGET_VARIATION > + m3back = m3back & " -freduce-target-variation" > + end > + end > return m3back > end > > diff --git a/m3-sys/m3back/src/M3C.m3 b/m3-sys/m3back/src/M3C.m3 > index f6740c7..22c6ade 100644 > --- a/m3-sys/m3back/src/M3C.m3 > +++ b/m3-sys/m3back/src/M3C.m3 > @@ -29,7 +29,7 @@ VAR CaseDefaultAssertFalse := FALSE; > > (* Taken together, these help debugging, as you get more lines in the > C and the error messages reference C line numbers *) > - CONST output_line_directives = TRUE; > + VAR output_line_directives := TRUE; > CONST output_extra_newlines = FALSE; > CONST inline_extract = FALSE; > > @@ -2197,7 +2197,9 @@ BEGIN > END; > > IF (*self.suppress_line_directive < 1 AND*) text_last_char = '\n' THEN > + IF NOT Target.ReduceTargetVariation THEN > Wr.PutText(self.c, self.line_directive); > + END; > self.width := 0; > self.last_char_was_newline := TRUE; > RETURN; > @@ -2205,7 +2207,9 @@ BEGIN > > IF Text.FindChar(text, '\n') # -1 THEN > self.width := 0; (* roughly *) > + IF NOT Target.ReduceTargetVariation THEN > Wr.PutText(self.c, self.nl_line_directive); > + END; > self.last_char_was_newline := TRUE; > RETURN; > END; > @@ -2217,11 +2221,13 @@ BEGIN > END; > > self.width := 0; > + IF NOT Target.ReduceTargetVariation THEN > IF self.last_char_was_newline THEN > Wr.PutText(self.c, self.line_directive); > ELSE > Wr.PutText(self.c, self.nl_line_directive); > END; > + END; > self.last_char_was_newline := TRUE; > END print; > > @@ -2339,9 +2345,10 @@ END set_error_handler; > PROCEDURE Prefix_Print(self: T; multipass: Multipass_t) = > BEGIN > self.comment("begin unit"); > + output_line_directives := output_line_directives AND NOT Target.ReduceTargetVariation; > + IF NOT Target.ReduceTargetVariation THEN > self.comment("M3_TARGET = ", Target.System_name); > - (* This is an unnecessary target-specific output. *) > - (* self.comment("M3_TARGET = ", Target.System_name); *) > + END; > self.comment("M3_WORDSIZE = ", IntToDec(Target.Word.size)); > self.static_link_id := M3ID.Add("_static_link"); > self.alloca_id := M3ID.Add("alloca"); > diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c > index dc52576..33f8844 100644 > --- a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c > +++ b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c > @@ -1065,6 +1065,9 @@ dbxout_init (const char *input_file_name) > labels. */ > ASM_GENERATE_INTERNAL_LABEL (ltext_label_name, "Ltext", 0); > > + /* Limit paths in debug output, to limit target variation. */ > + if (!reduce_target_variation) > + { > /* Put the current working directory in an N_SO symbol. */ > if (use_gnu_debug_info_extensions && !NO_DBX_MAIN_SOURCE_DIRECTORY) > { > @@ -1087,6 +1090,7 @@ dbxout_init (const char *input_file_name) > used_ltext_label_name = true; > #endif /* no DBX_OUTPUT_MAIN_SOURCE_DIRECTORY */ > } > + } > > mapped_name = remap_debug_filename (input_file_name); > #ifdef DBX_OUTPUT_MAIN_SOURCE_FILENAME > diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c > index 0da1021..0a48a65 100644 > --- a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c > +++ b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c > @@ -15450,19 +15450,20 @@ add_gnat_descriptive_type_attribute (dw_die_ref die, tree type, > static void > add_comp_dir_attribute (dw_die_ref die) > { > + /* Limit paths in debug output, to limit target variation. */ > + if (reduce_target_variation) > + return; > + > const char *wd = get_src_pwd (); > - char *wd1; > > if (wd == NULL) > return; > > if (DWARF2_DIR_SHOULD_END_WITH_SEPARATOR) > { > - int wdlen; > - > - wdlen = strlen (wd); > - wd1 = (char *) ggc_alloc_atomic (wdlen + 2); > - strcpy (wd1, wd); > + int const wdlen = (int)strlen (wd); > + char * const wd1 = (char *) ggc_alloc_atomic (wdlen + 2); > + memcpy (wd1, wd, wdlen); > wd1 [wdlen] = DIR_SEPARATOR; > wd1 [wdlen + 1] = 0; > wd = wd1; > diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c > index 63e4b92..d468632 100644 > --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c > +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c > @@ -217,11 +217,8 @@ const char * > get_src_pwd (void) > { > if (! src_pwd) > - { > - src_pwd = getpwd (); > - if (!src_pwd) > + if (reduce_target_variation || !(src_pwd = getpwd ())) > src_pwd = "."; > - } > > return src_pwd; > } > diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h > index 588cfdb..f4f7cc7 100644 > --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h > +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h > @@ -80,4 +80,6 @@ extern bool set_src_pwd (const char *); > extern HOST_WIDE_INT get_random_seed (bool); > extern const char *set_random_seed (const char *); > > +extern bool reduce_target_variation; > + > #endif /* ! GCC_TOPLEV_H */ > diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt > index 3bd0469..edfca96 100644 > --- a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt > +++ b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt > @@ -28,9 +28,6 @@ m3cg > Language > M3CG > > -y > -m3cg M3CG > - > fopcodes-trace > m3cg M3CG > Trace opcodes > @@ -59,10 +56,17 @@ ftypes-trace > m3cg M3CG > Trace types > > +freduce-target-variation > +m3cg M3CG > +Reduce target variation somewhat, such as by omitting current working > +directory from debug info. Many necessary target variations remain. > + > v > m3cg M3CG > +print version > > y > m3cg M3CG > +Trace opcodes > > ; This comment is to ensure we retain the blank line above. > diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c > index 03417ed..0cfa46a 100644 > --- a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c > +++ b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c > @@ -246,6 +246,7 @@ build_case_label (tree low_value, tree high_value, tree label_decl) > /*======================================================= OPTION HANDLING ===*/ > > static int option_trace_all; > +bool reduce_target_variation; > > /*===========================================================================*/ > > @@ -6364,6 +6365,10 @@ m3_handle_option (size_t code, PCSTR /*arg*/, int /*value*/) > case OPT_ftypes_trace: > option_trace_all += 1; > break; > + > + case OPT_freduce_target_variation: > + reduce_target_variation = true; > + break; > } > > return 1; > diff --git a/m3-sys/m3front/src/misc/Coverage.m3 b/m3-sys/m3front/src/misc/Coverage.m3 > index c04c902..73fff21 100644 > --- a/m3-sys/m3front/src/misc/Coverage.m3 > +++ b/m3-sys/m3front/src/misc/Coverage.m3 > @@ -77,8 +77,9 @@ PROCEDURE NoteProcedure (v: Value.T) = > PROCEDURE GenerateTables () = > VAR > nLines := MAX (0, maxLine - minLine) + 1; > + fname := Host.FileTail (Host.filename); > l_header := TLen (Header); > - l_fname := TLen (Host.filename); > + l_fname := TLen (fname); > l_trailer := TLen (Trailer); > size : INTEGER; > p : ProcHead; > @@ -124,10 +125,10 @@ PROCEDURE GenerateTables () = > (* CG.Init_int (size, Target.Integer.size, TInt.Zero, FALSE); *) > INC (size, Target.Integer.size); (*timestamp*) > > - CG.Init_intt (size, Target.Integer.size, Text.Length (Host.filename), FALSE); > + CG.Init_intt (size, Target.Integer.size, Text.Length (fname), FALSE); > INC (size, Target.Integer.size); (*fileLen*) > > - CG.Init_chars (size, Host.filename, FALSE); > + CG.Init_chars (size, fname, FALSE); > INC (size, l_fname * Target.Char.size); (*file*) > > CG.Init_intt (size, Target.Integer.size, minLine, FALSE); > diff --git a/m3-sys/m3front/src/misc/Host.i3 b/m3-sys/m3front/src/misc/Host.i3 > index c71489f..af6a89a 100644 > --- a/m3-sys/m3front/src/misc/Host.i3 > +++ b/m3-sys/m3front/src/misc/Host.i3 > @@ -75,4 +75,7 @@ PROCEDURE OpenUnit (name: M3ID.T; interface, generic: BOOLEAN; > > PROCEDURE CloseFile (rd: File.T); > > +PROCEDURE FileTail (path: TEXT): TEXT; > + (* returns the 'tail' of 'path' -- after any slashes or even spaces *) > + > END Host. > diff --git a/m3-sys/m3front/src/misc/Host.m3 b/m3-sys/m3front/src/misc/Host.m3 > index 962d3c6..79042e3 100644 > --- a/m3-sys/m3front/src/misc/Host.m3 > +++ b/m3-sys/m3front/src/misc/Host.m3 > @@ -9,7 +9,7 @@ > > MODULE Host; > > -IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler; > +IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler, Target; > > PROCEDURE Initialize (READONLY options: ARRAY OF TEXT): BOOLEAN = > BEGIN > @@ -192,5 +192,24 @@ PROCEDURE CloseFile (rd: File.T) = > END; > END CloseFile; > > +PROCEDURE FileTail (path: TEXT): TEXT = > + VAR c: CHAR; > + BEGIN > + IF NOT Target.ReduceTargetVariation THEN RETURN path; END; > + > + IF (path = NIL) THEN RETURN NIL END; > + > + (* search for the last slash or blank in the string *) > + FOR x := Text.Length (path) - 1 TO 0 BY -1 DO > + c := Text.GetChar (path, x); > + IF (c = '/') OR (c = ' ') OR (c = '\\') THEN > + RETURN Text.Sub (path, x+1); > + END; > + END; > + > + (* no slashes *) > + RETURN path; > + END FileTail; > + > BEGIN > END Host. > diff --git a/m3-sys/m3front/src/misc/M3Header.m3 b/m3-sys/m3front/src/misc/M3Header.m3 > index 1e4decf..877d77c 100644 > --- a/m3-sys/m3front/src/misc/M3Header.m3 > +++ b/m3-sys/m3front/src/misc/M3Header.m3 > @@ -104,7 +104,7 @@ PROCEDURE PushGeneric (VAR s: State) = > IF (s.generic = NIL) THEN s.failed := TRUE; RETURN; END; > > (* build a synthetic file name & start reading *) > - filename := old_filename & " => " & filename; > + filename := Host.FileTail(old_filename) & " => " & filename; > Scanner.Push (filename, s.generic, is_main := Scanner.in_main); > > (* make sure we got what we wanted *) > diff --git a/m3-sys/m3front/src/misc/Scanner.m3 b/m3-sys/m3front/src/misc/Scanner.m3 > index 7470374..e1dc024 100644 > --- a/m3-sys/m3front/src/misc/Scanner.m3 > +++ b/m3-sys/m3front/src/misc/Scanner.m3 > @@ -228,13 +228,16 @@ PROCEDURE Here (VAR file: TEXT; VAR line: INTEGER) = > BEGIN > file := files [offset DIV MaxLines]; > line := offset MOD MaxLines; > + IF Target.ReduceTargetVariation THEN > + file := Host.FileTail(file); > + END; > END Here; > > PROCEDURE LocalHere (VAR file: TEXT; VAR line: INTEGER) = > VAR fnum := offset DIV MaxLines; > BEGIN > IF (local_files[fnum] = NIL) THEN > - local_files[fnum] := files[fnum]; > + local_files[fnum] := Host.FileTail(files[fnum]); > END; > file := local_files [fnum]; > line := offset MOD MaxLines; > diff --git a/m3-sys/m3front/src/values/Module.m3 b/m3-sys/m3front/src/values/Module.m3 > index a085eab..576c857 100644 > --- a/m3-sys/m3front/src/values/Module.m3 > +++ b/m3-sys/m3front/src/values/Module.m3 > @@ -421,7 +421,7 @@ PROCEDURE PushGeneric (t: T; VAR rd: File.T): M3ID.T = > END; > > (* build a synthetic file name & start reading *) > - filename := old_filename & " => " & filename; > + filename := Host.FileTail(old_filename) & " => " & filename; > Scanner.Push (filename, rd, is_main := Scanner.in_main); > t.genericFile := filename; > > diff --git a/m3-sys/m3middle/src/Target.i3 b/m3-sys/m3middle/src/Target.i3 > index fe198d2..cd7acee 100644 > --- a/m3-sys/m3middle/src/Target.i3 > +++ b/m3-sys/m3middle/src/Target.i3 > @@ -529,4 +529,8 @@ VAR (*CONST*) > test for nested procedures passed as parameters must be more > elaborate (e.g. HPPA). *) > > + (* This removes some unnecessary target variation in the output, > + * such as current working directory in debug output. *) > + ReduceTargetVariation: BOOLEAN; > + > END Target. > diff --git a/scripts/python/pylib.py b/scripts/python/pylib.py > index 73e622f..487ad17 100755 > --- a/scripts/python/pylib.py > +++ b/scripts/python/pylib.py > @@ -388,7 +388,7 @@ def _GetAllTargets(): > > _CBackend = "c" in sys.argv or "C" in sys.argv > _BuildDirC = ["", "c"][_CBackend] > -_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why"] > +_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why", "reduce-target-variation", "reducetargetvariation"] > _SkipGccFlags = ["nogcc", "skipgcc", "omitgcc"] > _PossiblePylibFlags = ["noclean", "nocleangcc", "c", "C"] + _SkipGccFlags + _PossibleCm3Flags > > > Thank you, > - Jay > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Mon Jul 4 17:13:57 2016 From: jay.krell at cornell.edu (Jay K) Date: Mon, 4 Jul 2016 15:13:57 +0000 Subject: [M3devel] cm3 -reduce-target-variation / cm3cg -freduce-target-variation? In-Reply-To: <577A7998.1030907@lcwb.coop> References: , <577A7998.1030907@lcwb.coop> Message-ID: That is the intent. Some of this is actually a restoration of historical behavior -- except now still under a swtich, and arguably the historical behavior wasn't a good default. I'll remove the "f" on the cm3cg switch, so cm3 and cm3cg take the same parameters, which I think is a better convention than "-ffoo because cm3cg tends to use that." There is also a slight optimization in there and removal of a seemingly redundant cm3cg -v option -- it is specified twice. I'll check some build outputs to see if it redundant. ?- Jay ---------------------------------------- > Date: Mon, 4 Jul 2016 09:58:32 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com > Subject: Re: [M3devel] cm3 -reduce-target-variation / cm3cg -freduce-target-variation? > > > > On 07/04/2016 03:53 AM, Jay K wrote: >> I guess I should learn to use branches and/or pull requests. >> >> >> >> This adds cm3 -reduce-target-variation and cm3cg -freduce-target-variation. >> >> >> This meant for temporary development purposes. >> No user would use them. >> The point is to make e..g I386_FREEBSD and I386_NETBSD demonstrably identical, or at least closer to it. >> >> >> Ok to commit? >> > > If nothing happens unless these switches are explicitly supplied, it should > have no effect on anybody else, so OK. > >> >> Maybe name the two switches the same? Or something else? >> >> >> Maybe be less aggressive? >> i.e. even ../src/foo.m3 is getting changed to foo.m3. >> The real target is ../I386_DARWIN/foo.m3 => foo.m3 >> >> >> >> diff --git a/m3-sys/cm3/src/Main.m3 b/m3-sys/cm3/src/Main.m3 >> index 5d753e6..61b3f18 100644 >> --- a/m3-sys/cm3/src/Main.m3 >> +++ b/m3-sys/cm3/src/Main.m3 >> @@ -5,7 +5,7 @@ MODULE Main; >> >> IMPORT M3Timers, Pathname, Process, Quake; >> IMPORT RTCollector, RTParams, RTutils, Thread, Wr; >> -IMPORT TextTextTbl; >> +IMPORT TextTextTbl, Target; >> >> IMPORT Builder, Dirs, M3Build, M3Options, Makefile, Msg, Utils, WebFile; >> IMPORT MxConfig(*, M3Config, CMKey, CMCurrent *); >> @@ -18,6 +18,8 @@ VAR >> build_dir : TEXT := NIL; >> mach : Quake.Machine := NIL; >> >> +CONST BoolToText = ARRAY BOOLEAN OF TEXT{"FALSE", "TRUE"}; >> + >> PROCEDURE DefineIfNotDefined (qmachine: Quake.Machine; >> symbol, value: TEXT) RAISES {Quake.Error} = >> BEGIN >> @@ -73,6 +75,7 @@ VAR defs: TextTextTbl.T; >> (* DefineIfNotDefined (mach, "THREAD_LIBRARY", Version.ThreadLibrary); *) >> (* DefineIfNotDefined (mach, "WINDOW_LIBRARY", Version.WindowLibrary); *) >> DefineIfNotDefined (mach, "WORD_SIZE", MxConfig.HOST_WORD_SIZE); >> + DefineIfNotDefined (mach, "REDUCE_TARGET_VARIATION", BoolToText[Target.ReduceTargetVariation]); >> >> (* Even if the config file overrides the defaults, such as to do >> a cross build, the host characteristics are still available. *) >> diff --git a/m3-sys/cm3/src/Makefile.m3 b/m3-sys/cm3/src/Makefile.m3 >> index 64e169e..8b12320 100644 >> --- a/m3-sys/cm3/src/Makefile.m3 >> +++ b/m3-sys/cm3/src/Makefile.m3 >> @@ -5,7 +5,7 @@ MODULE Makefile; >> >> IMPORT FS, M3File, M3Timers, OSError, Params, Process, Text, Thread, Wr; >> IMPORT Arg, M3Build, M3Options, M3Path, Msg, Utils, TextSeq, TextTextTbl; >> -IMPORT MxConfig, Dirs, Version; >> +IMPORT MxConfig, Dirs, Version, Target; >> >> TYPE >> NK = M3Path.Kind; >> @@ -267,6 +267,9 @@ PROCEDURE ConvertOption (VAR s: State; arg: TEXT; arg_len: INTEGER) >> | 'r' => IF Text.Equal(arg, "-realclean") THEN >> ok := TRUE; (* mode set during the pre-scan *) >> s.found_work := TRUE; >> + ELSIF Text.Equal(arg, "-reduce-target-variation") THEN >> + ok := TRUE; >> + Target.ReduceTargetVariation := TRUE; >> END; >> >> | 's' => IF Text.Equal (arg, "-silent") THEN >> @@ -707,6 +710,9 @@ CONST >> " -group-writable \"", >> " -pb allow parallelism in running back-end (experimental)", >> " -no-m3ship-resolution use quake variables in .M3SHIP (experimental)", >> + " -reduce-target-variation omit target in some minor places such as", >> + " current working directory in debug information", >> + " for internal development purposes (showing target equivalence)", >> "", >> "environment variables:", >> " M3CONFIG platform dependent configuration file to use (cm3.cfg)", >> diff --git a/m3-sys/cminstall/src/config-no-install/cm3cfg.common b/m3-sys/cminstall/src/config-no-install/cm3cfg.common >> index 7334252..9806c3c 100644 >> --- a/m3-sys/cminstall/src/config-no-install/cm3cfg.common >> +++ b/m3-sys/cminstall/src/config-no-install/cm3cfg.common >> @@ -460,6 +460,12 @@ proc GetM3Back() is >> end end >> >> m3back = "@" & m3back & "cm3cg " & GetM3BackFlags() >> + >> + if defined ("REDUCE_TARGET_VARIATION") >> + if REDUCE_TARGET_VARIATION >> + m3back = m3back & " -freduce-target-variation" >> + end >> + end >> return m3back >> end >> >> diff --git a/m3-sys/m3back/src/M3C.m3 b/m3-sys/m3back/src/M3C.m3 >> index f6740c7..22c6ade 100644 >> --- a/m3-sys/m3back/src/M3C.m3 >> +++ b/m3-sys/m3back/src/M3C.m3 >> @@ -29,7 +29,7 @@ VAR CaseDefaultAssertFalse := FALSE; >> >> (* Taken together, these help debugging, as you get more lines in the >> C and the error messages reference C line numbers *) >> - CONST output_line_directives = TRUE; >> + VAR output_line_directives := TRUE; >> CONST output_extra_newlines = FALSE; >> CONST inline_extract = FALSE; >> >> @@ -2197,7 +2197,9 @@ BEGIN >> END; >> >> IF (*self.suppress_line_directive < 1 AND*) text_last_char = '\n' THEN >> + IF NOT Target.ReduceTargetVariation THEN >> Wr.PutText(self.c, self.line_directive); >> + END; >> self.width := 0; >> self.last_char_was_newline := TRUE; >> RETURN; >> @@ -2205,7 +2207,9 @@ BEGIN >> >> IF Text.FindChar(text, '\n') # -1 THEN >> self.width := 0; (* roughly *) >> + IF NOT Target.ReduceTargetVariation THEN >> Wr.PutText(self.c, self.nl_line_directive); >> + END; >> self.last_char_was_newline := TRUE; >> RETURN; >> END; >> @@ -2217,11 +2221,13 @@ BEGIN >> END; >> >> self.width := 0; >> + IF NOT Target.ReduceTargetVariation THEN >> IF self.last_char_was_newline THEN >> Wr.PutText(self.c, self.line_directive); >> ELSE >> Wr.PutText(self.c, self.nl_line_directive); >> END; >> + END; >> self.last_char_was_newline := TRUE; >> END print; >> >> @@ -2339,9 +2345,10 @@ END set_error_handler; >> PROCEDURE Prefix_Print(self: T; multipass: Multipass_t) = >> BEGIN >> self.comment("begin unit"); >> + output_line_directives := output_line_directives AND NOT Target.ReduceTargetVariation; >> + IF NOT Target.ReduceTargetVariation THEN >> self.comment("M3_TARGET = ", Target.System_name); >> - (* This is an unnecessary target-specific output. *) >> - (* self.comment("M3_TARGET = ", Target.System_name); *) >> + END; >> self.comment("M3_WORDSIZE = ", IntToDec(Target.Word.size)); >> self.static_link_id := M3ID.Add("_static_link"); >> self.alloca_id := M3ID.Add("alloca"); >> diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c >> index dc52576..33f8844 100644 >> --- a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c >> +++ b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c >> @@ -1065,6 +1065,9 @@ dbxout_init (const char *input_file_name) >> labels. */ >> ASM_GENERATE_INTERNAL_LABEL (ltext_label_name, "Ltext", 0); >> >> + /* Limit paths in debug output, to limit target variation. */ >> + if (!reduce_target_variation) >> + { >> /* Put the current working directory in an N_SO symbol. */ >> if (use_gnu_debug_info_extensions && !NO_DBX_MAIN_SOURCE_DIRECTORY) >> { >> @@ -1087,6 +1090,7 @@ dbxout_init (const char *input_file_name) >> used_ltext_label_name = true; >> #endif /* no DBX_OUTPUT_MAIN_SOURCE_DIRECTORY */ >> } >> + } >> >> mapped_name = remap_debug_filename (input_file_name); >> #ifdef DBX_OUTPUT_MAIN_SOURCE_FILENAME >> diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c >> index 0da1021..0a48a65 100644 >> --- a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c >> +++ b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c >> @@ -15450,19 +15450,20 @@ add_gnat_descriptive_type_attribute (dw_die_ref die, tree type, >> static void >> add_comp_dir_attribute (dw_die_ref die) >> { >> + /* Limit paths in debug output, to limit target variation. */ >> + if (reduce_target_variation) >> + return; >> + >> const char *wd = get_src_pwd (); >> - char *wd1; >> >> if (wd == NULL) >> return; >> >> if (DWARF2_DIR_SHOULD_END_WITH_SEPARATOR) >> { >> - int wdlen; >> - >> - wdlen = strlen (wd); >> - wd1 = (char *) ggc_alloc_atomic (wdlen + 2); >> - strcpy (wd1, wd); >> + int const wdlen = (int)strlen (wd); >> + char * const wd1 = (char *) ggc_alloc_atomic (wdlen + 2); >> + memcpy (wd1, wd, wdlen); >> wd1 [wdlen] = DIR_SEPARATOR; >> wd1 [wdlen + 1] = 0; >> wd = wd1; >> diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c >> index 63e4b92..d468632 100644 >> --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c >> +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c >> @@ -217,11 +217,8 @@ const char * >> get_src_pwd (void) >> { >> if (! src_pwd) >> - { >> - src_pwd = getpwd (); >> - if (!src_pwd) >> + if (reduce_target_variation || !(src_pwd = getpwd ())) >> src_pwd = "."; >> - } >> >> return src_pwd; >> } >> diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h >> index 588cfdb..f4f7cc7 100644 >> --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h >> +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h >> @@ -80,4 +80,6 @@ extern bool set_src_pwd (const char *); >> extern HOST_WIDE_INT get_random_seed (bool); >> extern const char *set_random_seed (const char *); >> >> +extern bool reduce_target_variation; >> + >> #endif /* ! GCC_TOPLEV_H */ >> diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt >> index 3bd0469..edfca96 100644 >> --- a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt >> +++ b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt >> @@ -28,9 +28,6 @@ m3cg >> Language >> M3CG >> >> -y >> -m3cg M3CG >> - >> fopcodes-trace >> m3cg M3CG >> Trace opcodes >> @@ -59,10 +56,17 @@ ftypes-trace >> m3cg M3CG >> Trace types >> >> +freduce-target-variation >> +m3cg M3CG >> +Reduce target variation somewhat, such as by omitting current working >> +directory from debug info. Many necessary target variations remain. >> + >> v >> m3cg M3CG >> +print version >> >> y >> m3cg M3CG >> +Trace opcodes >> >> ; This comment is to ensure we retain the blank line above. >> diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c >> index 03417ed..0cfa46a 100644 >> --- a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c >> +++ b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c >> @@ -246,6 +246,7 @@ build_case_label (tree low_value, tree high_value, tree label_decl) >> /*======================================================= OPTION HANDLING ===*/ >> >> static int option_trace_all; >> +bool reduce_target_variation; >> >> /*===========================================================================*/ >> >> @@ -6364,6 +6365,10 @@ m3_handle_option (size_t code, PCSTR /*arg*/, int /*value*/) >> case OPT_ftypes_trace: >> option_trace_all += 1; >> break; >> + >> + case OPT_freduce_target_variation: >> + reduce_target_variation = true; >> + break; >> } >> >> return 1; >> diff --git a/m3-sys/m3front/src/misc/Coverage.m3 b/m3-sys/m3front/src/misc/Coverage.m3 >> index c04c902..73fff21 100644 >> --- a/m3-sys/m3front/src/misc/Coverage.m3 >> +++ b/m3-sys/m3front/src/misc/Coverage.m3 >> @@ -77,8 +77,9 @@ PROCEDURE NoteProcedure (v: Value.T) = >> PROCEDURE GenerateTables () = >> VAR >> nLines := MAX (0, maxLine - minLine) + 1; >> + fname := Host.FileTail (Host.filename); >> l_header := TLen (Header); >> - l_fname := TLen (Host.filename); >> + l_fname := TLen (fname); >> l_trailer := TLen (Trailer); >> size : INTEGER; >> p : ProcHead; >> @@ -124,10 +125,10 @@ PROCEDURE GenerateTables () = >> (* CG.Init_int (size, Target.Integer.size, TInt.Zero, FALSE); *) >> INC (size, Target.Integer.size); (*timestamp*) >> >> - CG.Init_intt (size, Target.Integer.size, Text.Length (Host.filename), FALSE); >> + CG.Init_intt (size, Target.Integer.size, Text.Length (fname), FALSE); >> INC (size, Target.Integer.size); (*fileLen*) >> >> - CG.Init_chars (size, Host.filename, FALSE); >> + CG.Init_chars (size, fname, FALSE); >> INC (size, l_fname * Target.Char.size); (*file*) >> >> CG.Init_intt (size, Target.Integer.size, minLine, FALSE); >> diff --git a/m3-sys/m3front/src/misc/Host.i3 b/m3-sys/m3front/src/misc/Host.i3 >> index c71489f..af6a89a 100644 >> --- a/m3-sys/m3front/src/misc/Host.i3 >> +++ b/m3-sys/m3front/src/misc/Host.i3 >> @@ -75,4 +75,7 @@ PROCEDURE OpenUnit (name: M3ID.T; interface, generic: BOOLEAN; >> >> PROCEDURE CloseFile (rd: File.T); >> >> +PROCEDURE FileTail (path: TEXT): TEXT; >> + (* returns the 'tail' of 'path' -- after any slashes or even spaces *) >> + >> END Host. >> diff --git a/m3-sys/m3front/src/misc/Host.m3 b/m3-sys/m3front/src/misc/Host.m3 >> index 962d3c6..79042e3 100644 >> --- a/m3-sys/m3front/src/misc/Host.m3 >> +++ b/m3-sys/m3front/src/misc/Host.m3 >> @@ -9,7 +9,7 @@ >> >> MODULE Host; >> >> -IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler; >> +IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler, Target; >> >> PROCEDURE Initialize (READONLY options: ARRAY OF TEXT): BOOLEAN = >> BEGIN >> @@ -192,5 +192,24 @@ PROCEDURE CloseFile (rd: File.T) = >> END; >> END CloseFile; >> >> +PROCEDURE FileTail (path: TEXT): TEXT = >> + VAR c: CHAR; >> + BEGIN >> + IF NOT Target.ReduceTargetVariation THEN RETURN path; END; >> + >> + IF (path = NIL) THEN RETURN NIL END; >> + >> + (* search for the last slash or blank in the string *) >> + FOR x := Text.Length (path) - 1 TO 0 BY -1 DO >> + c := Text.GetChar (path, x); >> + IF (c = '/') OR (c = ' ') OR (c = '\\') THEN >> + RETURN Text.Sub (path, x+1); >> + END; >> + END; >> + >> + (* no slashes *) >> + RETURN path; >> + END FileTail; >> + >> BEGIN >> END Host. >> diff --git a/m3-sys/m3front/src/misc/M3Header.m3 b/m3-sys/m3front/src/misc/M3Header.m3 >> index 1e4decf..877d77c 100644 >> --- a/m3-sys/m3front/src/misc/M3Header.m3 >> +++ b/m3-sys/m3front/src/misc/M3Header.m3 >> @@ -104,7 +104,7 @@ PROCEDURE PushGeneric (VAR s: State) = >> IF (s.generic = NIL) THEN s.failed := TRUE; RETURN; END; >> >> (* build a synthetic file name & start reading *) >> - filename := old_filename & " => " & filename; >> + filename := Host.FileTail(old_filename) & " => " & filename; >> Scanner.Push (filename, s.generic, is_main := Scanner.in_main); >> >> (* make sure we got what we wanted *) >> diff --git a/m3-sys/m3front/src/misc/Scanner.m3 b/m3-sys/m3front/src/misc/Scanner.m3 >> index 7470374..e1dc024 100644 >> --- a/m3-sys/m3front/src/misc/Scanner.m3 >> +++ b/m3-sys/m3front/src/misc/Scanner.m3 >> @@ -228,13 +228,16 @@ PROCEDURE Here (VAR file: TEXT; VAR line: INTEGER) = >> BEGIN >> file := files [offset DIV MaxLines]; >> line := offset MOD MaxLines; >> + IF Target.ReduceTargetVariation THEN >> + file := Host.FileTail(file); >> + END; >> END Here; >> >> PROCEDURE LocalHere (VAR file: TEXT; VAR line: INTEGER) = >> VAR fnum := offset DIV MaxLines; >> BEGIN >> IF (local_files[fnum] = NIL) THEN >> - local_files[fnum] := files[fnum]; >> + local_files[fnum] := Host.FileTail(files[fnum]); >> END; >> file := local_files [fnum]; >> line := offset MOD MaxLines; >> diff --git a/m3-sys/m3front/src/values/Module.m3 b/m3-sys/m3front/src/values/Module.m3 >> index a085eab..576c857 100644 >> --- a/m3-sys/m3front/src/values/Module.m3 >> +++ b/m3-sys/m3front/src/values/Module.m3 >> @@ -421,7 +421,7 @@ PROCEDURE PushGeneric (t: T; VAR rd: File.T): M3ID.T = >> END; >> >> (* build a synthetic file name & start reading *) >> - filename := old_filename & " => " & filename; >> + filename := Host.FileTail(old_filename) & " => " & filename; >> Scanner.Push (filename, rd, is_main := Scanner.in_main); >> t.genericFile := filename; >> >> diff --git a/m3-sys/m3middle/src/Target.i3 b/m3-sys/m3middle/src/Target.i3 >> index fe198d2..cd7acee 100644 >> --- a/m3-sys/m3middle/src/Target.i3 >> +++ b/m3-sys/m3middle/src/Target.i3 >> @@ -529,4 +529,8 @@ VAR (*CONST*) >> test for nested procedures passed as parameters must be more >> elaborate (e.g. HPPA). *) >> >> + (* This removes some unnecessary target variation in the output, >> + * such as current working directory in debug output. *) >> + ReduceTargetVariation: BOOLEAN; >> + >> END Target. >> diff --git a/scripts/python/pylib.py b/scripts/python/pylib.py >> index 73e622f..487ad17 100755 >> --- a/scripts/python/pylib.py >> +++ b/scripts/python/pylib.py >> @@ -388,7 +388,7 @@ def _GetAllTargets(): >> >> _CBackend = "c" in sys.argv or "C" in sys.argv >> _BuildDirC = ["", "c"][_CBackend] >> -_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why"] >> +_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why", "reduce-target-variation", "reducetargetvariation"] >> _SkipGccFlags = ["nogcc", "skipgcc", "omitgcc"] >> _PossiblePylibFlags = ["noclean", "nocleangcc", "c", "C"] + _SkipGccFlags + _PossibleCm3Flags >> >> >> Thank you, >> - Jay >> >> >> _______________________________________________ >> M3devel mailing list >> M3devel at elegosoft.com >> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> > > -- > Rodney Bates > rodney.m.bates at acm.org > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel From jay.krell at cornell.edu Mon Jul 4 23:17:23 2016 From: jay.krell at cornell.edu (Jay K) Date: Mon, 4 Jul 2016 21:17:23 +0000 Subject: [M3devel] defaulting to parallel backend? In-Reply-To: <577A78EA.5000600@lcwb.coop> References: , <577A78EA.5000600@lcwb.coop> Message-ID: ok. Presumably something like: name it to match C++11 They call it: unsigned std::thread::hardware_concurrency(void); We should call it: PROCEDURE Thread.HardwareConcurrency(): INTEGER; OR PROCEDURE Thread.HardwareConcurrency(): Word.T; One could specify the following result for "unknown": <0, 0, 1 1 is nice and safe, but lossy vs. actually knowing it is 1. One could add another function for the known-ness boolean: PROCEDURE Thread.HardwareConcurrencyValid(): BOOLEAN; or PROCEDURE Thread.HardwareConcurrencyKnown(): BOOLEAN; "Unknown" I don't believe would really stem from the underlying implementation, there is no error path in the underlying operating systems, but be for platforms we don't have a port for, so is less interesting and just returning a sub-par 1 isn't so bad. ?- Jay ---------------------------------------- > Date: Mon, 4 Jul 2016 09:55:38 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com > Subject: Re: [M3devel] defaulting to parallel backend? > > I prefer to use one fewer than the number of processors, so I can > web browse, read email, review code, etc. while waiting for a > compile. The compiler can saturate them all, making any other > stuff very slow. > > On 07/04/2016 03:45 AM, Jay K wrote: >> I suggest we make parallel backend the default. >> >> You can turn it off with -pb 1. >> >> Turn it "up" with -pb >> >> And the default is a thread per processor. >> This will be kernel specific code, but we can get far with >> >> 1) Win32 GetSystemInfo, if applicable (e.g. C backend) >> 2) sysconf(_SC_NPROCESSORS_ONLN) >> >> >> and I'll research the various OSes (Darwin, Linux, *BSD, Solaris, and possibly AIX, HP-UX, Irix, Mingw, Cygwin). >> ok? >> >> I could really use the speed boost and multi-processor machines are overwhelmingly common now (though easily avoided with VMs). >> >> >> - Jay >> >> >> _______________________________________________ >> M3devel mailing list >> M3devel at elegosoft.com >> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> > > -- > Rodney Bates > rodney.m.bates at acm.org > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel From jay.krell at cornell.edu Tue Jul 5 09:54:14 2016 From: jay.krell at cornell.edu (Jay K) Date: Tue, 5 Jul 2016 07:54:14 +0000 Subject: [M3devel] some code from boost? Message-ID: I'd like to bring in some code from boost. To m3core -- Thread.HardwareConcurrency, i.e. the number of processors It is licensed like so: Boost Software License - Version 1.0 - August 17th, 2003 Permission is hereby granted, free of charge, to any person or organization obtaining a copy of the software and accompanying documentation covered by this license (the "Software") to use, reproduce, display, distribute, execute, and transmit the Software, and to prepare derivative works of the Software, and to permit third-parties to whom the Software is furnished to do so, all subject to the following: The copyright notices in the Software and this entire statement, including the above license grant, this restriction and the following disclaimer, must be included in all copies of the Software, in whole or in part, and all derivative works of the Software, unless such copies or derivative works are solely in the form of machine-executable object code generated by a source language processor. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. http://www.boost.org/LICENSE_1_0.txt) I find the license a little long and would rather not have more, but it is a good body of work overall. I would put the code in circa m3core/thread/boost/win32, m3core/thread/boost/posix or maybe m3core/thread/Common/ThreadBoostWin32.c ThreadBoostPosix.c ok? esp. on the licensing? It is just a few lines of code. They call this thread::hardware_concurrency and it is the number of CPUs or cores (this number is actually ambiguous these days, with packages, cores, and hyper-threading). They have another number called physical_concurrency we might want to also provide. They have implementations for at least the following the systems: ?Win32 GetSystemInfo ?glibc?get_nprocs ?sysconf(_SC_NPROCESSORS_ONLN) ?Apple and FreeBSD?sysctlbyname("hw.ncpu") ?HP-UX?pthread_num_processors_np For licensing or at least credit purposs, it should be two files -- Win32 and Posix. The code is C++ but can be changed to C, and for should should be. I suppose, if the license is too restrictive, I can keep it out of m3core and put it only in cm3? ?- Jay From jay.krell at cornell.edu Tue Jul 5 10:19:23 2016 From: jay.krell at cornell.edu (Jay K) Date: Tue, 5 Jul 2016 08:19:23 +0000 Subject: [M3devel] purported condition variable problems on Win32? Message-ID: https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV vs. http://www.cs.wustl.edu/~schmidt/win32-cv-1.html vs. https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain I wrote this: PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; alertable: BOOLEAN) RAISES {Alerted} = (* LL = m on entry and exit, but not for the duration * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp * NOTE that they merge the user lock and the condition lock. * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html * "3.3. The Generation Count Solution" *) Do we have the problems described in README.CV? I haven't looked through the ACE code to see to what extent they resemble solution 3.3, or if they changed as a result of this discussion -- which I admit I don't understandand haven't read closely. Spurious wakeups are ok, though should be minimized. I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. Thank you, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Jul 5 10:26:36 2016 From: jay.krell at cornell.edu (Jay K) Date: Tue, 5 Jul 2016 08:26:36 +0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: Message-ID: Here is another implementation to consider: https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Tue, 5 Jul 2016 08:19:23 +0000 Subject: [M3devel] purported condition variable problems on Win32? https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV vs. http://www.cs.wustl.edu/~schmidt/win32-cv-1.html vs. https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain I wrote this: PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; alertable: BOOLEAN) RAISES {Alerted} = (* LL = m on entry and exit, but not for the duration * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp * NOTE that they merge the user lock and the condition lock. * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html * "3.3. The Generation Count Solution" *) Do we have the problems described in README.CV? I haven't looked through the ACE code to see to what extent they resemble solution 3.3, or if they changed as a result of this discussion -- which I admit I don't understandand haven't read closely. Spurious wakeups are ok, though should be minimized. I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. Thank you, - Jay _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From lemming at henning-thielemann.de Tue Jul 5 18:34:54 2016 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Tue, 5 Jul 2016 18:34:54 +0200 (CEST) Subject: [M3devel] defaulting to parallel backend? In-Reply-To: References: Message-ID: On Mon, 4 Jul 2016, Jay K wrote: > I suggest we make parallel backend the default. Generally I do not like if a program eats up all computing power without being told so. E.g. my machine has 8 hyperthreads but only 4 cores. Running on 8 processes hardly gives maximum efficiency, most oftenly speed is maximal at 3 processes. I also like the idea to use the remaining computing power for different tasks. Please just leave the choice of the number of processes to the user. From lemming at henning-thielemann.de Tue Jul 5 18:36:45 2016 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Tue, 5 Jul 2016 18:36:45 +0200 (CEST) Subject: [M3devel] defaulting to parallel backend? In-Reply-To: <577A78EA.5000600@lcwb.coop> References: <577A78EA.5000600@lcwb.coop> Message-ID: On Mon, 4 Jul 2016, Rodney M. Bates wrote: > I prefer to use one fewer than the number of processors, so I can > web browse, read email, review code, etc. while waiting for a > compile. The compiler can saturate them all, making any other > stuff very slow. Slowdown can happen even earlier if the compilation is bound by disk access time. From jay.krell at cornell.edu Wed Jul 6 07:44:45 2016 From: jay.krell at cornell.edu (Jay K) Date: Wed, 6 Jul 2016 05:44:45 +0000 Subject: [M3devel] defaulting to parallel backend? In-Reply-To: References: , <577A78EA.5000600@lcwb.coop>, Message-ID: There is no good easy answer here. You really want everything on the computer to use all of the available resources, to service whatever the human wants done as fast as possible, while remaining responsive e.g. to requests to cancel or requests to increase the work load. This is largely the job of the kernel, but it is an impossible job. To some extent we are obligated to over subscribe, in order to provide the kernel more information. But, again, it is an impossible job. Besides it probably being NP complete to complete the submitted work as soon as possible, we also don't know the user's priorities. I could propose "pb 1" to mean run single threaded, putting the burden on you to give. But that is a little rude. Note that historically you didn't have as much flexibility here -- single core systems. Anyway, how about the default be half? Or 1 if <= 3 cores available? I seem to have 4 cores, so this will help me. I understand this is still not great, e.g. if you have 10 high priority single processors jobs, and one low priority Modula-3 build, Modula-3 will by default use more than you wish. And yes, I understand disk are another resource. And network -- disk can be on network. You want to operate at the "edge of resource exhaustion" in general. - Jay > Date: Tue, 5 Jul 2016 18:36:45 +0200 > From: lemming at henning-thielemann.de > To: rodney.m.bates at acm.org > Subject: Re: [M3devel] defaulting to parallel backend? > CC: m3devel at elegosoft.com > > > On Mon, 4 Jul 2016, Rodney M. Bates wrote: > > > I prefer to use one fewer than the number of processors, so I can > > web browse, read email, review code, etc. while waiting for a > > compile. The compiler can saturate them all, making any other > > stuff very slow. > > Slowdown can happen even earlier if the compilation is bound by disk > access time. > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Jul 7 11:24:08 2016 From: jay.krell at cornell.edu (Jay K) Date: Thu, 7 Jul 2016 09:24:08 +0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: , Message-ID: So...I do NOT understand all of this. However, comparing the three implementationsand attempting to understand ours,I am struck by the following - Our broadcast seems ok, but - our signal seems to wake everyone, and then...they race a bit, one will decide it is last, and reset the event, but every prior waiter is still woken. Why not merge the following chunks: WHILE (NOT alerted) AND (NOT waitDone) DO... 1 EnterCriticalSection(conditionLock); waitDone := (c.tickets # 0 AND c.counter # count); LeaveCriticalSection(conditionLock); END; (* WHILE *) IF waitDone THEN alerted := FALSE; END; m.acquire(); 2 EnterCriticalSection(conditionLock); DEC(c.waiters); IF waitDone THEN esp. here. DEC(c.tickets); lastWaiter := (c.tickets = 0); END; LeaveCriticalSection(conditionLock); That is, if we decrement tickets earlier within the first critical section, while we will still wake everyone, only one will decide waitDone and the rest will keep looping. A downside of this, perhaps, is that waking all for Broadcast might be a little slower. but more so, in both the "ptw32" (pthreads for win32) and Boost threads implementations, they seem to deal with this differently than us, and they each do about the same thing -- they use a counted semaphore. In the boost case, it appears they duplicate the semaphore for every notification generation, which seems expensive. Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. Or they can do their own reference counting? jdk7 seems to looke like jdk6. The code is gone in jdk8 and I can't easily find the delete in history. The lock merging jdk does probably helps here too. In fact they merge #1 and #2 above as a result. But they still initially wake all threads for signal, not just broadcast. There is also the problem that all the event waits are followed by EnterCriticalSection (or jdk facsimile). - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: RE: [M3devel] purported condition variable problems on Win32? Date: Tue, 5 Jul 2016 08:26:36 +0000 Here is another implementation to consider: https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Tue, 5 Jul 2016 08:19:23 +0000 Subject: [M3devel] purported condition variable problems on Win32? https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV vs. http://www.cs.wustl.edu/~schmidt/win32-cv-1.html vs. https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain I wrote this: PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; alertable: BOOLEAN) RAISES {Alerted} = (* LL = m on entry and exit, but not for the duration * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp * NOTE that they merge the user lock and the condition lock. * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html * "3.3. The Generation Count Solution" *) Do we have the problems described in README.CV? I haven't looked through the ACE code to see to what extent they resemble solution 3.3, or if they changed as a result of this discussion -- which I admit I don't understandand haven't read closely. Spurious wakeups are ok, though should be minimized. I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. Thank you, - Jay _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Wed Jul 13 20:26:59 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Wed, 13 Jul 2016 13:26:59 -0500 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: Message-ID: <578687F3.1010108@lcwb.coop> I've begun looking at this. On 07/05/2016 03:19 AM, Jay K wrote: > https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > As for spurious wakeups, the M3 Thread.i3 documentation of Signal explicitly allows this: "One or more threads waiting on c become eligible to run", whereas the posix pthread_cond_signal, according to schmidt, "notifies one thread waiting on condition variable cv". As I recall, these (the M3 semantics) agree with the original Brinch-Hansen condition variables. So, for implementing posix CVs, this would be a true bug, but not so for M3 Thread CVs. > vs. > > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > vs. > https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > I wrote this: > PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > alertable: BOOLEAN) RAISES {Alerted} = > (* LL = m on entry and exit, but not for the duration > * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > * NOTE that they merge the user lock and the condition lock. > * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > * "3.3. The Generation Count Solution" > *) > > Do we have the problems described in README.CV? > > I haven't looked through the ACE code to see > to what extent they resemble solution 3.3, or if they > changed as a result of this discussion -- which I admit I don't understand > and haven't read closely. > > > Spurious wakeups are ok, though should be minimized. > > I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > Thank you, > - Jay > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Sun Jul 17 23:06:19 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 17 Jul 2016 21:06:19 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: , Message-ID: <578BF33B.2030605@lcwb.coop> I spent some time looking over the signal part of ThreadWin32.m3. A few comments: This looks like a very direct implementation of Schmidts' generation count algorithm, with a number of important little details additionally taken care of. On small difference is at line 308 (B, below) where Schmidt does c.counter>count instead of #. Usually, this would make no difference, since counter only increases, but on a long-running program, it could overflow. If this happened,as it often does, as silent wrap-around to FIRST(INTEGER), some threads could be trapped for an extremely long time, waiting for counter come back around greater than their count. As coded in ThreadWin32, they would be allowed to proceed normally. The only glitch would be that, if there were simultaneously waiting threads separated by NUMBER(INTEGER) truly different generations (but the same value of c.counter), the recent ones could unfairly proceed along with the extremely old ones. That seems extremely less likely than the mere existence of an overflow. Both type Condition and type Activation have a field waitEvent: HANDLE. Condition.waitEvent is extensively used, but Activation.waitEvent is only created at :537 and deleted at :554, but never used in any real way. It could be deleted. As for this code: EnterCriticalSection(conditionLock); (* Capture the value of the counter before we start waiting. * We will not stop waiting until the counter changes. * That is, we will not stop waiting until a signal * comes in after we start waiting. *) count := c.counter; INC(c.waiters); LeaveCriticalSection(conditionLock); m.release(); (* can this be moved to before taking conditionLock? *) No, it is important to sample and save c.counter in the order threads wait. Retaining m until this is done ensures this. Otherwise, some other thread that waited later could have altered c.counter before this one gets conditionLock and saves its copy of c.counter. I have made several comment changes to ThreadWin32.m3 that would have made it easier to vet. I will commit these, but only comments. It is either very difficult or impossible for me to test or even recompile this module, and, especially as fragile as this kind of code is, I don't want to commit any substantive changes untested and uncompiled. More below: On 07/07/2016 04:24 AM, Jay K wrote: > So...I do NOT understand all of this. > > > However, comparing the three implementations > and attempting to understand ours, > I am struck by the following > > - Our broadcast seems ok, but > - our signal seems to wake everyone, > and then...they race a bit, > one will decide it is last, and > reset the event, but every prior waiter > is still woken. > Why not merge the following chunks: I don't think this will work. It is important that, once a thread has decided, at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, if it is also the last of its generation to proceed, resets c.waitevent. This is ensured by its having already reacquired m first, preventing any other thread from returning from its Wait yet. And it can't attempt to acquire m while already holding conditionLock, because this would invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while already holding m, at line 265. > > > WHILE (NOT alerted) AND (NOT waitDone) DO > . > . > . > 1 > EnterCriticalSection(conditionLock); > waitDone := (c.tickets # 0 AND c.counter # count); A:---------^ B:--------------------------------------------------^ > LeaveCriticalSection(conditionLock); > END; (* WHILE *) > IF waitDone THEN > alerted := FALSE; > END; > m.acquire(); > 2 > EnterCriticalSection(conditionLock); > DEC(c.waiters); > IF waitDone THEN esp. here. > DEC(c.tickets); C:---------------^ > lastWaiter := (c.tickets = 0); > END; > LeaveCriticalSection(conditionLock); > > > That is, if we decrement tickets earlier > within the first critical section, > while we will still wake everyone, > only one will decide waitDone and the rest will keep looping. > A downside of this, perhaps, is that waking all for Broadcast > might be a little slower. > but more so, in both the "ptw32" (pthreads for win32) and Boost threads > implementations, they seem to deal with this differently than us, > and they each do about the same thing -- they use a counted semaphore. > In the boost case, it appears they duplicate the semaphore for every > notification generation, which seems expensive. > > Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > Or they can do their own reference counting? > jdk7 seems to looke like jdk6. > The code is gone in jdk8 and I can't easily find the delete in history. > The lock merging jdk does probably helps here too. > In fact they merge #1 and #2 above as a result. > But they still initially wake all threads for signal, not just broadcast. > There is also the problem that all the event waits > are followed by EnterCriticalSection (or jdk facsimile). > - Jay > > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Subject: RE: [M3devel] purported condition variable problems on Win32? > Date: Tue, 5 Jul 2016 08:26:36 +0000 > > Here is another implementation to consider: > > https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > > - Jay > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Tue, 5 Jul 2016 08:19:23 +0000 > Subject: [M3devel] purported condition variable problems on Win32? > > https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > > vs. > > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > vs. > https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > I wrote this: > PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > alertable: BOOLEAN) RAISES {Alerted} = > (* LL = m on entry and exit, but not for the duration > * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > * NOTE that they merge the user lock and the condition lock. > * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > * "3.3. The Generation Count Solution" > *) > > Do we have the problems described in README.CV? > > I haven't looked through the ACE code to see > to what extent they resemble solution 3.3, or if they > changed as a result of this discussion -- which I admit I don't understand > and haven't read closely. > > > Spurious wakeups are ok, though should be minimized. > > I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > Thank you, > - Jay > > _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Sun Jul 17 23:23:50 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 17 Jul 2016 21:23:50 -0000 Subject: [M3devel] defaulting to parallel backend? In-Reply-To: References: , <577A78EA.5000600@lcwb.coop>, Message-ID: <578BF74F.1020009@lcwb.coop> For systems that have it, how about just using the 'nice' command? On 07/06/2016 12:44 AM, Jay K wrote: > There is no good easy answer here. > > > You really want everything on the computer to use all of the > available resources, to service whatever the human wants done > as fast as possible, while remaining responsive e.g. to > requests to cancel or requests to increase the work load. > > > This is largely the job of the kernel, but it is an impossible job. > > To some extent we are obligated to over subscribe, in order > to provide the kernel more information. > But, again, it is an impossible job. > > Besides it probably being NP complete to complete the submitted work > as soon as possible, we also don't know the user's priorities. > > I could propose "pb 1" to mean run single threaded, putting > the burden on you to give. But that is a little rude. > > Note that historically you didn't have as much flexibility here -- single core systems. > > > Anyway, how about the default be half? > Or 1 if <= 3 cores available? > I seem to have 4 cores, so this will help me. > > > I understand this is still not great, e.g. if you have 10 > high priority single processors jobs, and one low priority > Modula-3 build, Modula-3 will by default use more than you wish. > > And yes, I understand disk are another resource. > And network -- disk can be on network. > You want to operate at the "edge of resource exhaustion" in general. > > > - Jay > > > > > > Date: Tue, 5 Jul 2016 18:36:45 +0200 > > From: lemming at henning-thielemann.de > > To: rodney.m.bates at acm.org > > Subject: Re: [M3devel] defaulting to parallel backend? > > CC: m3devel at elegosoft.com > > > > > > On Mon, 4 Jul 2016, Rodney M. Bates wrote: > > > > > I prefer to use one fewer than the number of processors, so I can > > > web browse, read email, review code, etc. while waiting for a > > > compile. The compiler can saturate them all, making any other > > > stuff very slow. > > > > Slowdown can happen even earlier if the compilation is bound by disk > > access time. > > _______________________________________________ > > M3devel mailing list > > M3devel at elegosoft.com > > https://m3lists.elegosoft.com/mailman/listinfo/m3devel -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Mon Jul 18 16:26:28 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 18 Jul 2016 14:26:28 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: <578BF33B.2030605@lcwb.coop> References: <578BF33B.2030605@lcwb.coop> Message-ID: <578CE6C7.2010007@lcwb.coop> I spent some time looking over the signal part of ThreadWin32.m3. A few comments: This looks like a very direct implementation of Schmidts' generation count algorithm, with a number of important little details additionally taken care of. On small difference is at line 308 (B, below) where Schmidt does c.counter>count instead of #. Usually, this would make no difference, since counter only increases, but on a long-running program, it could overflow. If this happened,as it often does, as silent wrap-around to FIRST(INTEGER), some threads could be trapped for an extremely long time, waiting for counter come back around greater than their count. As coded in ThreadWin32, they would be allowed to proceed normally. The only glitch would be that, if there were simultaneously waiting threads separated by NUMBER(INTEGER) truly different generations (but the same value of c.counter), the recent ones could unfairly proceed along with the extremely old ones. That seems extremely less likely than the mere existence of an overflow. Both type Condition and type Activation have a field waitEvent: HANDLE. Condition.waitEvent is extensively used, but Activation.waitEvent is only created at :537 and deleted at :554, but never used in any real way. It could be deleted. As for this code: EnterCriticalSection(conditionLock); (* Capture the value of the counter before we start waiting. * We will not stop waiting until the counter changes. * That is, we will not stop waiting until a signal * comes in after we start waiting. *) count := c.counter; INC(c.waiters); LeaveCriticalSection(conditionLock); m.release(); (* can this be moved to before taking conditionLock? *) No, it is important to sample and save c.counter in the order threads wait. Retaining m until this is done ensures this. Otherwise, some other thread that waited later could have altered c.counter before this one gets conditionLock and saves its copy of c.counter. I have made several comment changes to ThreadWin32.m3 that would have made it easier to vet. I will commit these, but only comments. It is either very difficult or impossible for me to test or even recompile this module, and, especially as fragile as this kind of code is, I don't want to commit any substantive changes untested and uncompiled. More below: On 07/07/2016 04:24 AM, Jay K wrote: > So...I do NOT understand all of this. > > > However, comparing the three implementations > and attempting to understand ours, > I am struck by the following > > - Our broadcast seems ok, but > - our signal seems to wake everyone, > and then...they race a bit, > one will decide it is last, and > reset the event, but every prior waiter > is still woken. > Why not merge the following chunks: I don't think this will work. It is important that, once a thread has decided, at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, if it is also the last of its generation to proceed, resets c.waitevent. This is ensured by its having already reacquired m first, preventing any other thread from returning from its Wait yet. And it can't attempt to acquire m while already holding conditionLock, because this would invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while already holding m, at line 265. > > > WHILE (NOT alerted) AND (NOT waitDone) DO > . > . > . > 1 > EnterCriticalSection(conditionLock); > waitDone := (c.tickets # 0 AND c.counter # count); A:---------^ B:--------------------------------------------------^ > LeaveCriticalSection(conditionLock); > END; (* WHILE *) > IF waitDone THEN > alerted := FALSE; > END; > m.acquire(); > 2 > EnterCriticalSection(conditionLock); > DEC(c.waiters); > IF waitDone THEN esp. here. > DEC(c.tickets); C:---------------^ > lastWaiter := (c.tickets = 0); > END; > LeaveCriticalSection(conditionLock); > > > That is, if we decrement tickets earlier > within the first critical section, > while we will still wake everyone, > only one will decide waitDone and the rest will keep looping. > A downside of this, perhaps, is that waking all for Broadcast > might be a little slower. > but more so, in both the "ptw32" (pthreads for win32) and Boost threads > implementations, they seem to deal with this differently than us, > and they each do about the same thing -- they use a counted semaphore. > In the boost case, it appears they duplicate the semaphore for every > notification generation, which seems expensive. > > Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > Or they can do their own reference counting? > jdk7 seems to looke like jdk6. > The code is gone in jdk8 and I can't easily find the delete in history. > The lock merging jdk does probably helps here too. > In fact they merge #1 and #2 above as a result. > But they still initially wake all threads for signal, not just broadcast. > There is also the problem that all the event waits > are followed by EnterCriticalSection (or jdk facsimile). > - Jay > > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Subject: RE: [M3devel] purported condition variable problems on Win32? > Date: Tue, 5 Jul 2016 08:26:36 +0000 > > Here is another implementation to consider: > > https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > > - Jay > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Tue, 5 Jul 2016 08:19:23 +0000 > Subject: [M3devel] purported condition variable problems on Win32? > > https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > > vs. > > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > vs. > https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > I wrote this: > PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > alertable: BOOLEAN) RAISES {Alerted} = > (* LL = m on entry and exit, but not for the duration > * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > * NOTE that they merge the user lock and the condition lock. > * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > * "3.3. The Generation Count Solution" > *) > > Do we have the problems described in README.CV? > > I haven't looked through the ACE code to see > to what extent they resemble solution 3.3, or if they > changed as a result of this discussion -- which I admit I don't understand > and haven't read closely. > > > Spurious wakeups are ok, though should be minimized. > > I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > Thank you, > - Jay > > _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Tue Jul 19 03:53:24 2016 From: jay.krell at cornell.edu (Jay) Date: Tue, 19 Jul 2016 01:53:24 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: <578CE6C7.2010007@lcwb.coop> References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop> Message-ID: <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. Otherwise previous version I think had a very large lock. Changing the < to !=, is that really ok? I get that 32bit overflow is possible. We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. I guess you are saying it would be unfair, which is ok. There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. I'll check on the unused event. - Jay > On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > > > > > > I spent some time looking over the signal part of ThreadWin32.m3. A few > comments: > > This looks like a very direct implementation of Schmidts' generation count > algorithm, with a number of important little details additionally taken care > of. > > On small difference is at line 308 (B, below) where Schmidt does > c.counter>count instead of #. Usually, this would make no difference, > since counter only increases, but on a long-running program, it could > overflow. If this happened,as it often does, as silent wrap-around to > FIRST(INTEGER), some threads could be trapped for an extremely long > time, waiting for counter come back around greater than their count. As > coded in ThreadWin32, they would be allowed to proceed normally. > > The only glitch would be that, if there were simultaneously waiting threads > separated by NUMBER(INTEGER) truly different generations (but the same value > of c.counter), the recent ones could unfairly proceed along with the > extremely old ones. That seems extremely less likely than the > mere existence of an overflow. > > Both type Condition and type Activation have a field waitEvent: HANDLE. > Condition.waitEvent is extensively used, but Activation.waitEvent is > only created at :537 and deleted at :554, but never used in any real > way. It could be deleted. > > As for this code: > > EnterCriticalSection(conditionLock); > > (* Capture the value of the counter before we start waiting. > * We will not stop waiting until the counter changes. > * That is, we will not stop waiting until a signal > * comes in after we start waiting. > *) > > count := c.counter; > INC(c.waiters); > > LeaveCriticalSection(conditionLock); > m.release(); (* can this be moved to before taking conditionLock? *) > > No, it is important to sample and save c.counter in the order threads wait. > Retaining m until this is done ensures this. Otherwise, some other thread > that waited later could have altered c.counter before this one gets conditionLock > and saves its copy of c.counter. > > I have made several comment changes to ThreadWin32.m3 that would have made > it easier to vet. I will commit these, but only comments. It is either > very difficult or impossible for me to test or even recompile this module, > and, especially as fragile as this kind of code is, I don't want to commit > any substantive changes untested and uncompiled. > > More below: > >> On 07/07/2016 04:24 AM, Jay K wrote: >> So...I do NOT understand all of this. >> >> >> However, comparing the three implementations >> and attempting to understand ours, >> I am struck by the following >> >> - Our broadcast seems ok, but >> - our signal seems to wake everyone, >> and then...they race a bit, >> one will decide it is last, and >> reset the event, but every prior waiter >> is still woken. >> Why not merge the following chunks: > > I don't think this will work. It is important that, once a thread has decided, > at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > if it is also the last of its generation to proceed, resets c.waitevent. This > is ensured by its having already reacquired m first, preventing any other thread > from returning from its Wait yet. > > And it can't attempt to acquire m while already holding conditionLock, because this would > invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > already holding m, at line 265. > >> >> >> WHILE (NOT alerted) AND (NOT waitDone) DO >> . >> . >> . >> 1 >> EnterCriticalSection(conditionLock); >> waitDone := (c.tickets # 0 AND c.counter # count); > A:---------^ > B:--------------------------------------------------^ >> LeaveCriticalSection(conditionLock); >> END; (* WHILE *) >> IF waitDone THEN >> alerted := FALSE; >> END; >> m.acquire(); >> 2 >> EnterCriticalSection(conditionLock); >> DEC(c.waiters); >> IF waitDone THEN esp. here. >> DEC(c.tickets); > > C:---------------^ >> lastWaiter := (c.tickets = 0); >> END; >> LeaveCriticalSection(conditionLock); >> >> >> That is, if we decrement tickets earlier >> within the first critical section, >> while we will still wake everyone, >> only one will decide waitDone and the rest will keep looping. >> A downside of this, perhaps, is that waking all for Broadcast >> might be a little slower. >> but more so, in both the "ptw32" (pthreads for win32) and Boost threads >> implementations, they seem to deal with this differently than us, >> and they each do about the same thing -- they use a counted semaphore. >> In the boost case, it appears they duplicate the semaphore for every >> notification generation, which seems expensive. >> >> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. >> Or they can do their own reference counting? >> jdk7 seems to looke like jdk6. >> The code is gone in jdk8 and I can't easily find the delete in history. >> The lock merging jdk does probably helps here too. >> In fact they merge #1 and #2 above as a result. >> But they still initially wake all threads for signal, not just broadcast. >> There is also the problem that all the event waits >> are followed by EnterCriticalSection (or jdk facsimile). >> - Jay >> >> >> >> >> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --- >> From: jay.krell at cornell.edu >> To: m3devel at elegosoft.com >> Subject: RE: [M3devel] purported condition variable problems on Win32? >> Date: Tue, 5 Jul 2016 08:26:36 +0000 >> >> Here is another implementation to consider: >> >> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp >> >> - Jay >> >> >> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --- >> From: jay.krell at cornell.edu >> To: m3devel at elegosoft.com >> Date: Tue, 5 Jul 2016 08:19:23 +0000 >> Subject: [M3devel] purported condition variable problems on Win32? >> >> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV >> >> vs. >> >> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >> vs. >> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain >> I wrote this: >> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; >> alertable: BOOLEAN) RAISES {Alerted} = >> (* LL = m on entry and exit, but not for the duration >> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp >> * NOTE that they merge the user lock and the condition lock. >> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >> * "3.3. The Generation Count Solution" >> *) >> >> Do we have the problems described in README.CV? >> >> I haven't looked through the ACE code to see >> to what extent they resemble solution 3.3, or if they >> changed as a result of this discussion -- which I admit I don't understand >> and haven't read closely. >> >> >> Spurious wakeups are ok, though should be minimized. >> >> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. >> >> Thank you, >> - Jay >> >> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> >> >> _______________________________________________ >> M3devel mailing list >> M3devel at elegosoft.com >> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > -- > Rodney Bates > rodney.m.bates at acm.org > > From rodney_bates at lcwb.coop Wed Jul 20 18:44:24 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Wed, 20 Jul 2016 16:44:24 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop> <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> Message-ID: <578FAA2F.6020105@lcwb.coop> In looking at ThreadWin32.m3 more, I think I see a significant bug. This gets confusing, since there are Modula-3 waits, signals, etc. and similar concepts in Win32, but which are not the same. I am going the try to put "M3-" and "Win-" prefixes on things to make this clearer. I cartainly need to do this for my own benefit, if for nobody else. Suppose two threads are both M3-waiting in Wait(m,c,...), further Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. c.tickets = 0, and c.waiters = 2. Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), which will allow both the waiters to Win-wakeup sometime soon. Before releasing c.lock, Signal increments c.tickets to 1 (supposedly to eventually allow only one waiter to M3-wakeup) and increments c.counter, setting things up so that the waitDone expression at :308 will be TRUE for the waiting threads, the next time they evaluate it. Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, releases c.lock, and tries to acquire m. The problem arises because the second waiter can get to these steps before the first can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The second to reach :308 will find c.tickets # 0, as did the first, and proceed. Each will eventually, do a M3-wake. If this were Posix condition variables, this would be wrong, because Posix says only one waiter proceeds. For M3, it's OK, if unnecessary, because M3 says one or more waiters proceed. However, each will, in the process, get to :322, and DEC(c.tickets). Signal only provided one ticket, but the waiters have taken two. c.tickets goes negative. After this, when some thread can do another (M3-)Wait, and some other does another (M3-)Signal, c.tickets increments only back to zero, which means the new waiter will not M3-wake, even though it should. I am going to try to construct a test case that will force this scenario. See more below: On 07/18/2016 08:53 PM, Jay wrote: > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > Otherwise previous version I think had a very large lock. > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > I guess you are saying it would be unfair, which is ok. > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > I'll check on the unused event. > > - Jay > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: >> >> >> >> >> >> I spent some time looking over the signal part of ThreadWin32.m3. A few >> comments: >> >> This looks like a very direct implementation of Schmidts' generation count >> algorithm, with a number of important little details additionally taken care >> of. >> >> On small difference is at line 308 (B, below) where Schmidt does >> c.counter>count instead of #. Usually, this would make no difference, >> since counter only increases, but on a long-running program, it could >> overflow. If this happened,as it often does, as silent wrap-around to >> FIRST(INTEGER), some threads could be trapped for an extremely long >> time, waiting for counter come back around greater than their count. As >> coded in ThreadWin32, they would be allowed to proceed normally. >> >> The only glitch would be that, if there were simultaneously waiting threads >> separated by NUMBER(INTEGER) truly different generations (but the same value >> of c.counter), the recent ones could unfairly proceed along with the >> extremely old ones. That seems extremely less likely than the >> mere existence of an overflow. >> >> Both type Condition and type Activation have a field waitEvent: HANDLE. >> Condition.waitEvent is extensively used, but Activation.waitEvent is >> only created at :537 and deleted at :554, but never used in any real >> way. It could be deleted. >> >> As for this code: >> >> EnterCriticalSection(conditionLock); >> >> (* Capture the value of the counter before we start waiting. >> * We will not stop waiting until the counter changes. >> * That is, we will not stop waiting until a signal >> * comes in after we start waiting. >> *) >> >> count := c.counter; >> INC(c.waiters); >> >> LeaveCriticalSection(conditionLock); >> m.release(); (* can this be moved to before taking conditionLock? *) >> >> No, it is important to sample and save c.counter in the order threads wait. >> Retaining m until this is done ensures this. Otherwise, some other thread >> that waited later could have altered c.counter before this one gets conditionLock >> and saves its copy of c.counter. >> >> I have made several comment changes to ThreadWin32.m3 that would have made >> it easier to vet. I will commit these, but only comments. It is either >> very difficult or impossible for me to test or even recompile this module, >> and, especially as fragile as this kind of code is, I don't want to commit >> any substantive changes untested and uncompiled. >> >> More below: >> >>> On 07/07/2016 04:24 AM, Jay K wrote: >>> So...I do NOT understand all of this. >>> >>> >>> However, comparing the three implementations >>> and attempting to understand ours, >>> I am struck by the following >>> >>> - Our broadcast seems ok, but >>> - our signal seems to wake everyone, >>> and then...they race a bit, >>> one will decide it is last, and >>> reset the event, but every prior waiter >>> is still woken. >>> Why not merge the following chunks: >> >> I don't think this will work. It is important that, once a thread has decided, >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, >> if it is also the last of its generation to proceed, resets c.waitevent. This >> is ensured by its having already reacquired m first, preventing any other thread >> from returning from its Wait yet. I was wrong about this. There is already no fairness being enforced here. Even with only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is arbitrary anyway. There is no benefit in trying to control the order they reacquire m. c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. Merging, as you proposed, is probably close to the fix for this bug. >> >> And it can't attempt to acquire m while already holding conditionLock, because this would >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while >> already holding m, at line 265. >> >>> >>> >>> WHILE (NOT alerted) AND (NOT waitDone) DO >>> . >>> . >>> . >>> 1 >>> EnterCriticalSection(conditionLock); >>> waitDone := (c.tickets # 0 AND c.counter # count); >> A:---------^ >> B:--------------------------------------------------^ >>> LeaveCriticalSection(conditionLock); >>> END; (* WHILE *) >>> IF waitDone THEN >>> alerted := FALSE; >>> END; >>> m.acquire(); >>> 2 >>> EnterCriticalSection(conditionLock); >>> DEC(c.waiters); >>> IF waitDone THEN esp. here. >>> DEC(c.tickets); >> >> C:---------------^ >>> lastWaiter := (c.tickets = 0); >>> END; >>> LeaveCriticalSection(conditionLock); >>> >>> >>> That is, if we decrement tickets earlier >>> within the first critical section, >>> while we will still wake everyone, >>> only one will decide waitDone and the rest will keep looping. >>> A downside of this, perhaps, is that waking all for Broadcast >>> might be a little slower. >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads >>> implementations, they seem to deal with this differently than us, >>> and they each do about the same thing -- they use a counted semaphore. >>> In the boost case, it appears they duplicate the semaphore for every >>> notification generation, which seems expensive. >>> >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. >>> Or they can do their own reference counting? >>> jdk7 seems to looke like jdk6. >>> The code is gone in jdk8 and I can't easily find the delete in history. >>> The lock merging jdk does probably helps here too. >>> In fact they merge #1 and #2 above as a result. >>> But they still initially wake all threads for signal, not just broadcast. >>> There is also the problem that all the event waits >>> are followed by EnterCriticalSection (or jdk facsimile). >>> - Jay >>> >>> >>> >>> >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> --- >>> From: jay.krell at cornell.edu >>> To: m3devel at elegosoft.com >>> Subject: RE: [M3devel] purported condition variable problems on Win32? >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 >>> >>> Here is another implementation to consider: >>> >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp >>> >>> - Jay >>> >>> >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> --- >>> From: jay.krell at cornell.edu >>> To: m3devel at elegosoft.com >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 >>> Subject: [M3devel] purported condition variable problems on Win32? >>> >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV >>> >>> vs. >>> >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >>> vs. >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain >>> I wrote this: >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; >>> alertable: BOOLEAN) RAISES {Alerted} = >>> (* LL = m on entry and exit, but not for the duration >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp >>> * NOTE that they merge the user lock and the condition lock. >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >>> * "3.3. The Generation Count Solution" >>> *) >>> >>> Do we have the problems described in README.CV? >>> >>> I haven't looked through the ACE code to see >>> to what extent they resemble solution 3.3, or if they >>> changed as a result of this discussion -- which I admit I don't understand >>> and haven't read closely. >>> >>> >>> Spurious wakeups are ok, though should be minimized. >>> >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. >>> >>> Thank you, >>> - Jay >>> >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>> >>> >>> _______________________________________________ >>> M3devel mailing list >>> M3devel at elegosoft.com >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> >> -- >> Rodney Bates >> rodney.m.bates at acm.org >> >> > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Wed Jul 20 20:00:28 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Wed, 20 Jul 2016 18:00:28 -0000 Subject: [M3devel] Yet another bug in Win-32 generation count threads Message-ID: <578FBC29.3000308@lcwb.coop> The more I look at this generation count approach to implementing Modula-3 condition variables on top of what Windows has, the less I like it. It's very hard to understand, but as I get closer, I think I see yet another bug. Assume ticket counting is fixed. There are several waiting threads, say 5. c.waiters = 5 and c.counter = 0. A M3-Broadcast happens. It does (Win-)SetEvent(c.waitEvent), sets c.counter := 1, and c.tickets := 5, making all the waiters eligible to get past the waitDone test, which they start doing. From the abstract M3-condition variable point of view, all have already been released from the Condition variable, but have some cleanup to do inside (M3-)Wait. One by one, they proceed, say 3 of them. Two waiters remain, c.waiters = c.tickets = 2, and all have counts that make them eligible for waitDone. Now, two more waiters do M3-Waits. c.counter still = 1, and their local counts = 1, making them ineligible to get past waitDone. Then a M3-Signal happens. c.counter gets incremented to 2, so all all 4 waiters are now eligible to pass that test. c.tickets gets increased to 3, supposedly so the two that were broadcast-released and one of the new ones can proceed. But it is randomly possible that both the new ones get released (which would be OK), taking two of the tickets, one of the older ones get released, taking the last ticket, and the remaining is stuck waiting when it should not, for lack of a ticket. -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Thu Jul 21 04:00:50 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Thu, 21 Jul 2016 02:00:50 -0000 Subject: [M3devel] Yet another bug in Win-32 generation count threads In-Reply-To: <578FBC29.3000308@lcwb.coop> References: <578FBC29.3000308@lcwb.coop> Message-ID: <57902C9E.1000005@lcwb.coop> If I were to try to fix the Thread condition variables on Windows, I would consider a scheme entirely different from any of those given in Schmidt. Associate the Win-event that a M3-Wait operation uses with a thread, not a condition variable. Since a thread can wait on at most one condition at a time, the Win-event will have at most one waiting thread. This also means allocation of the Win-events is easy: put one in each Activation (which is one-to-one with a thread). (Perhaps the unused waitEvent in an activation was left over from some variant of this Scheme?) Put the Activations of the multiple threads waiting on a Condition in some kind of linked list rooted at the Condition, with link pointers in the Activations. This makes it easy to code almost any wakeup fairness policy in a single place, with single-threaded code: removal from this list, rather than complex interactions among a signaller and many waiter threads. Or, alternatively, it could be done when inserting into the list. Simple FIFO is the obvious choice, but anything could be easily implemented., such as thread priorities. Making the order of reacquisition of the mutex match the order of SetEvent would require more complex work, but shouldn't be too hard. It is an easy-to-maintain invariant that a waiter is waiting on a Win-event iff its Activation is on the list of some condition. This means a signaller will always know a Win-event it is about to call SetEvent on has a thread waiting in it, so can do a self- resetting SetEvent and never worry that the event will remain signalled afterwards. With these restrictions on their use, Win-events behave just like condition variables, greatly simplifying things. All the solutions Schmidt describes suffer from problems originating in having muliple waiters Win-waiting on a single Win-event. One detail that could require some thought is a waiting thread's Activation needs to be removed from its list when the thread is destroyed. On 07/20/2016 01:00 PM, Rodney M. Bates wrote: > The more I look at this generation count approach to implementing > Modula-3 condition variables on top of what Windows has, the less > I like it. It's very hard to understand, but as I get closer, > I think I see yet another bug. > > Assume ticket counting is fixed. > > There are several waiting threads, say 5. c.waiters = 5 and > c.counter = 0. A M3-Broadcast happens. It does > (Win-)SetEvent(c.waitEvent), sets c.counter := 1, and > c.tickets := 5, making all the waiters eligible to get > past the waitDone test, which they start doing. From the > abstract M3-condition variable point of view, all have > already been released from the Condition variable, but > have some cleanup to do inside (M3-)Wait. > > One by one, they proceed, say 3 of them. Two waiters > remain, c.waiters = c.tickets = 2, and all have counts > that make them eligible for waitDone. > > Now, two more waiters do M3-Waits. c.counter still = 1, and > their local counts = 1, making them ineligible to get > past waitDone. Then a M3-Signal happens. c.counter > gets incremented to 2, so all all 4 waiters are now eligible > to pass that test. c.tickets gets increased to 3, supposedly > so the two that were broadcast-released and one of the new > ones can proceed. > > But it is randomly possible that both the new ones get > released (which would be OK), taking two of the tickets, > one of the older ones get released, taking the last ticket, > and the remaining is stuck waiting when it should not, for > lack of a ticket. > -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Thu Jul 21 08:35:45 2016 From: jay.krell at cornell.edu (Jay K) Date: Thu, 21 Jul 2016 06:35:45 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: <578FAA2F.6020105@lcwb.coop> References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop> Message-ID: Am I understanding this? There are some waiters. There is a broadcast ("wake all"). Some waiters are woken. There are more waiters. There is a signal ("wake one"). There is a chance that some of the original waiters will remain waiting while the later waiters are not. That is, the signaled threads kind of steal the wake intended by the broadcast. Is this unfairness or incorrectness? It feels like unfairness. It feels like condition variables are specified fairly weakly. - Jay > Date: Wed, 20 Jul 2016 11:43:27 -0500 > From: rodney_bates at lcwb.coop > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] purported condition variable problems on Win32? > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > This gets confusing, since there are Modula-3 waits, signals, > etc. and similar concepts in Win32, but which are not the > same. I am going the try to put "M3-" and "Win-" prefixes on > things to make this clearer. I cartainly need to do this > for my own benefit, if for nobody else. > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > c.tickets = 0, and c.waiters = 2. > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > which will allow both the waiters to Win-wakeup sometime soon. > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > to eventually allow only one waiter to M3-wakeup) and increments > c.counter, setting things up so that the waitDone expression at :308 > will be TRUE for the waiting threads, the next time they evaluate it. > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > releases c.lock, and tries to acquire m. The problem arises > because the second waiter can get to these steps before the first > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > second to reach :308 will find c.tickets # 0, as did the first, > and proceed. > > Each will eventually, do a M3-wake. If this were Posix condition > variables, this would be wrong, because Posix says only one > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > says one or more waiters proceed. However, each will, in the > process, get to :322, and DEC(c.tickets). Signal only provided > one ticket, but the waiters have taken two. c.tickets goes > negative. > > After this, when some thread can do another (M3-)Wait, and some > other does another (M3-)Signal, c.tickets increments only back to > zero, which means the new waiter will not M3-wake, even though it > should. > > > I am going to try to construct a test case that will force > this scenario. > > See more below: > > > On 07/18/2016 08:53 PM, Jay wrote: > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > Otherwise previous version I think had a very large lock. > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > I guess you are saying it would be unfair, which is ok. > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > I'll check on the unused event. > > > > - Jay > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > >> > >> > >> > >> > >> > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > >> comments: > >> > >> This looks like a very direct implementation of Schmidts' generation count > >> algorithm, with a number of important little details additionally taken care > >> of. > >> > >> On small difference is at line 308 (B, below) where Schmidt does > >> c.counter>count instead of #. Usually, this would make no difference, > >> since counter only increases, but on a long-running program, it could > >> overflow. If this happened,as it often does, as silent wrap-around to > >> FIRST(INTEGER), some threads could be trapped for an extremely long > >> time, waiting for counter come back around greater than their count. As > >> coded in ThreadWin32, they would be allowed to proceed normally. > >> > >> The only glitch would be that, if there were simultaneously waiting threads > >> separated by NUMBER(INTEGER) truly different generations (but the same value > >> of c.counter), the recent ones could unfairly proceed along with the > >> extremely old ones. That seems extremely less likely than the > >> mere existence of an overflow. > >> > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > >> only created at :537 and deleted at :554, but never used in any real > >> way. It could be deleted. > >> > >> As for this code: > >> > >> EnterCriticalSection(conditionLock); > >> > >> (* Capture the value of the counter before we start waiting. > >> * We will not stop waiting until the counter changes. > >> * That is, we will not stop waiting until a signal > >> * comes in after we start waiting. > >> *) > >> > >> count := c.counter; > >> INC(c.waiters); > >> > >> LeaveCriticalSection(conditionLock); > >> m.release(); (* can this be moved to before taking conditionLock? *) > >> > >> No, it is important to sample and save c.counter in the order threads wait. > >> Retaining m until this is done ensures this. Otherwise, some other thread > >> that waited later could have altered c.counter before this one gets conditionLock > >> and saves its copy of c.counter. > >> > >> I have made several comment changes to ThreadWin32.m3 that would have made > >> it easier to vet. I will commit these, but only comments. It is either > >> very difficult or impossible for me to test or even recompile this module, > >> and, especially as fragile as this kind of code is, I don't want to commit > >> any substantive changes untested and uncompiled. > >> > >> More below: > >> > >>> On 07/07/2016 04:24 AM, Jay K wrote: > >>> So...I do NOT understand all of this. > >>> > >>> > >>> However, comparing the three implementations > >>> and attempting to understand ours, > >>> I am struck by the following > >>> > >>> - Our broadcast seems ok, but > >>> - our signal seems to wake everyone, > >>> and then...they race a bit, > >>> one will decide it is last, and > >>> reset the event, but every prior waiter > >>> is still woken. > >>> Why not merge the following chunks: > >> > >> I don't think this will work. It is important that, once a thread has decided, > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > >> if it is also the last of its generation to proceed, resets c.waitevent. This > >> is ensured by its having already reacquired m first, preventing any other thread > >> from returning from its Wait yet. > > I was wrong about this. There is already no fairness being enforced here. Even with > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > Merging, as you proposed, is probably close to the fix for this bug. > > > >> > >> And it can't attempt to acquire m while already holding conditionLock, because this would > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > >> already holding m, at line 265. > >> > >>> > >>> > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > >>> . > >>> . > >>> . > >>> 1 > >>> EnterCriticalSection(conditionLock); > >>> waitDone := (c.tickets # 0 AND c.counter # count); > >> A:---------^ > >> B:--------------------------------------------------^ > >>> LeaveCriticalSection(conditionLock); > >>> END; (* WHILE *) > >>> IF waitDone THEN > >>> alerted := FALSE; > >>> END; > >>> m.acquire(); > >>> 2 > >>> EnterCriticalSection(conditionLock); > >>> DEC(c.waiters); > >>> IF waitDone THEN esp. here. > >>> DEC(c.tickets); > >> > >> C:---------------^ > >>> lastWaiter := (c.tickets = 0); > >>> END; > >>> LeaveCriticalSection(conditionLock); > >>> > >>> > >>> That is, if we decrement tickets earlier > >>> within the first critical section, > >>> while we will still wake everyone, > >>> only one will decide waitDone and the rest will keep looping. > >>> A downside of this, perhaps, is that waking all for Broadcast > >>> might be a little slower. > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > >>> implementations, they seem to deal with this differently than us, > >>> and they each do about the same thing -- they use a counted semaphore. > >>> In the boost case, it appears they duplicate the semaphore for every > >>> notification generation, which seems expensive. > >>> > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > >>> Or they can do their own reference counting? > >>> jdk7 seems to looke like jdk6. > >>> The code is gone in jdk8 and I can't easily find the delete in history. > >>> The lock merging jdk does probably helps here too. > >>> In fact they merge #1 and #2 above as a result. > >>> But they still initially wake all threads for signal, not just broadcast. > >>> There is also the problem that all the event waits > >>> are followed by EnterCriticalSection (or jdk facsimile). > >>> - Jay > >>> > >>> > >>> > >>> > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! > >> --- > >>> From: jay.krell at cornell.edu > >>> To: m3devel at elegosoft.com > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > >>> > >>> Here is another implementation to consider: > >>> > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > >>> > >>> - Jay > >>> > >>> > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! > >> --- > >>> From: jay.krell at cornell.edu > >>> To: m3devel at elegosoft.com > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > >>> Subject: [M3devel] purported condition variable problems on Win32? > >>> > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > >>> > >>> vs. > >>> > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >>> vs. > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > >>> I wrote this: > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > >>> alertable: BOOLEAN) RAISES {Alerted} = > >>> (* LL = m on entry and exit, but not for the duration > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > >>> * NOTE that they merge the user lock and the condition lock. > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >>> * "3.3. The Generation Count Solution" > >>> *) > >>> > >>> Do we have the problems described in README.CV? > >>> > >>> I haven't looked through the ACE code to see > >>> to what extent they resemble solution 3.3, or if they > >>> changed as a result of this discussion -- which I admit I don't understand > >>> and haven't read closely. > >>> > >>> > >>> Spurious wakeups are ok, though should be minimized. > >>> > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > >>> > >>> Thank you, > >>> - Jay > >>> > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >>> > >>> > >>> _______________________________________________ > >>> M3devel mailing list > >>> M3devel at elegosoft.com > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >> > >> -- > >> Rodney Bates > >> rodney.m.bates at acm.org > >> > >> > > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Wed Jul 20 17:36:13 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Wed, 20 Jul 2016 15:36:13 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop> <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> Message-ID: <578F99C5.3010702@lcwb.coop> On 07/18/2016 08:53 PM, Jay wrote: > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > Otherwise previous version I think had a very large lock. > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > I guess you are saying it would be unfair, which is ok. I think ths is OK as coded. The # will, overwhelmingly often, make wraparound work as expected, whereas the > would leave threads trapped at the upper end for ~ FIRST(INTEGER) generations. I am not sure about the cases that have problems with #, but for anything to happen at all, there would have to be a wrap, further followed by a thread that got starved for another ~ NUMBER(INTEGER) generations of other threads getting through. Even at 32 bits, this is extremely unlikely, and probably something else would fail before this. If it really were a problem, I think counter can be reset to zero any time there are no waiters, which would usually be very often. > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > I'll check on the unused event. > > - Jay > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: >> >> >> >> >> >> I spent some time looking over the signal part of ThreadWin32.m3. A few >> comments: >> >> This looks like a very direct implementation of Schmidts' generation count >> algorithm, with a number of important little details additionally taken care >> of. >> >> On small difference is at line 308 (B, below) where Schmidt does >> c.counter>count instead of #. Usually, this would make no difference, >> since counter only increases, but on a long-running program, it could >> overflow. If this happened,as it often does, as silent wrap-around to >> FIRST(INTEGER), some threads could be trapped for an extremely long >> time, waiting for counter come back around greater than their count. As >> coded in ThreadWin32, they would be allowed to proceed normally. >> >> The only glitch would be that, if there were simultaneously waiting threads >> separated by NUMBER(INTEGER) truly different generations (but the same value >> of c.counter), the recent ones could unfairly proceed along with the >> extremely old ones. That seems extremely less likely than the >> mere existence of an overflow. >> >> Both type Condition and type Activation have a field waitEvent: HANDLE. >> Condition.waitEvent is extensively used, but Activation.waitEvent is >> only created at :537 and deleted at :554, but never used in any real >> way. It could be deleted. >> >> As for this code: >> >> EnterCriticalSection(conditionLock); >> >> (* Capture the value of the counter before we start waiting. >> * We will not stop waiting until the counter changes. >> * That is, we will not stop waiting until a signal >> * comes in after we start waiting. >> *) >> >> count := c.counter; >> INC(c.waiters); >> >> LeaveCriticalSection(conditionLock); >> m.release(); (* can this be moved to before taking conditionLock? *) >> >> No, it is important to sample and save c.counter in the order threads wait. >> Retaining m until this is done ensures this. Otherwise, some other thread >> that waited later could have altered c.counter before this one gets conditionLock >> and saves its copy of c.counter. >> >> I have made several comment changes to ThreadWin32.m3 that would have made >> it easier to vet. I will commit these, but only comments. It is either >> very difficult or impossible for me to test or even recompile this module, >> and, especially as fragile as this kind of code is, I don't want to commit >> any substantive changes untested and uncompiled. >> >> More below: >> >>> On 07/07/2016 04:24 AM, Jay K wrote: >>> So...I do NOT understand all of this. >>> >>> >>> However, comparing the three implementations >>> and attempting to understand ours, >>> I am struck by the following >>> >>> - Our broadcast seems ok, but >>> - our signal seems to wake everyone, >>> and then...they race a bit, >>> one will decide it is last, and >>> reset the event, but every prior waiter >>> is still woken. >>> Why not merge the following chunks: >> >> I don't think this will work. It is important that, once a thread has decided, >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, >> if it is also the last of its generation to proceed, resets c.waitevent. This >> is ensured by its having already reacquired m first, preventing any other thread >> from returning from its Wait yet. >> >> And it can't attempt to acquire m while already holding conditionLock, because this would >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while >> already holding m, at line 265. >> >>> >>> >>> WHILE (NOT alerted) AND (NOT waitDone) DO >>> . >>> . >>> . >>> 1 >>> EnterCriticalSection(conditionLock); >>> waitDone := (c.tickets # 0 AND c.counter # count); >> A:---------^ >> B:--------------------------------------------------^ >>> LeaveCriticalSection(conditionLock); >>> END; (* WHILE *) >>> IF waitDone THEN >>> alerted := FALSE; >>> END; >>> m.acquire(); >>> 2 >>> EnterCriticalSection(conditionLock); >>> DEC(c.waiters); >>> IF waitDone THEN esp. here. >>> DEC(c.tickets); >> >> C:---------------^ >>> lastWaiter := (c.tickets = 0); >>> END; >>> LeaveCriticalSection(conditionLock); >>> >>> >>> That is, if we decrement tickets earlier >>> within the first critical section, >>> while we will still wake everyone, >>> only one will decide waitDone and the rest will keep looping. >>> A downside of this, perhaps, is that waking all for Broadcast >>> might be a little slower. >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads >>> implementations, they seem to deal with this differently than us, >>> and they each do about the same thing -- they use a counted semaphore. >>> In the boost case, it appears they duplicate the semaphore for every >>> notification generation, which seems expensive. >>> >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. >>> Or they can do their own reference counting? >>> jdk7 seems to looke like jdk6. >>> The code is gone in jdk8 and I can't easily find the delete in history. >>> The lock merging jdk does probably helps here too. >>> In fact they merge #1 and #2 above as a result. >>> But they still initially wake all threads for signal, not just broadcast. >>> There is also the problem that all the event waits >>> are followed by EnterCriticalSection (or jdk facsimile). >>> - Jay >>> >>> >>> >>> >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> --- >>> From: jay.krell at cornell.edu >>> To: m3devel at elegosoft.com >>> Subject: RE: [M3devel] purported condition variable problems on Win32? >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 >>> >>> Here is another implementation to consider: >>> >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp >>> >>> - Jay >>> >>> >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> --- >>> From: jay.krell at cornell.edu >>> To: m3devel at elegosoft.com >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 >>> Subject: [M3devel] purported condition variable problems on Win32? >>> >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV >>> >>> vs. >>> >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >>> vs. >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain >>> I wrote this: >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; >>> alertable: BOOLEAN) RAISES {Alerted} = >>> (* LL = m on entry and exit, but not for the duration >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp >>> * NOTE that they merge the user lock and the condition lock. >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >>> * "3.3. The Generation Count Solution" >>> *) >>> >>> Do we have the problems described in README.CV? >>> >>> I haven't looked through the ACE code to see >>> to what extent they resemble solution 3.3, or if they >>> changed as a result of this discussion -- which I admit I don't understand >>> and haven't read closely. >>> >>> >>> Spurious wakeups are ok, though should be minimized. >>> >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. >>> >>> Thank you, >>> - Jay >>> >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>> >>> >>> _______________________________________________ >>> M3devel mailing list >>> M3devel at elegosoft.com >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> >> -- >> Rodney Bates >> rodney.m.bates at acm.org >> >> > -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Thu Jul 21 08:43:20 2016 From: jay.krell at cornell.edu (Jay K) Date: Thu, 21 Jul 2016 06:43:20 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop>, <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop>, Message-ID: That question was meant for the other bug. Can this one be fixed by rechecking if tickets is positive before decrementing it? This seems like a big problem in Schmidt's code. I'm nervous about merging the locks, but maybe. In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. Though I realize that is actually necessarily part of what gets done. Schmidt clearly did not merge the locks, and the Java version does. The Java version also has its own simple and not-terrible critical section. The Java version also disappeared in newer releases and I didn't track down what happened to it. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org CC: m3devel at elegosoft.com Subject: RE: [M3devel] purported condition variable problems on Win32? Date: Thu, 21 Jul 2016 06:35:41 +0000 Am I understanding this? There are some waiters. There is a broadcast ("wake all"). Some waiters are woken. There are more waiters. There is a signal ("wake one"). There is a chance that some of the original waiters will remain waiting while the later waiters are not. That is, the signaled threads kind of steal the wake intended by the broadcast. Is this unfairness or incorrectness? It feels like unfairness. It feels like condition variables are specified fairly weakly. - Jay > Date: Wed, 20 Jul 2016 11:43:27 -0500 > From: rodney_bates at lcwb.coop > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] purported condition variable problems on Win32? > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > This gets confusing, since there are Modula-3 waits, signals, > etc. and similar concepts in Win32, but which are not the > same. I am going the try to put "M3-" and "Win-" prefixes on > things to make this clearer. I cartainly need to do this > for my own benefit, if for nobody else. > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > c.tickets = 0, and c.waiters = 2. > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > which will allow both the waiters to Win-wakeup sometime soon. > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > to eventually allow only one waiter to M3-wakeup) and increments > c.counter, setting things up so that the waitDone expression at :308 > will be TRUE for the waiting threads, the next time they evaluate it. > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > releases c.lock, and tries to acquire m. The problem arises > because the second waiter can get to these steps before the first > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > second to reach :308 will find c.tickets # 0, as did the first, > and proceed. > > Each will eventually, do a M3-wake. If this were Posix condition > variables, this would be wrong, because Posix says only one > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > says one or more waiters proceed. However, each will, in the > process, get to :322, and DEC(c.tickets). Signal only provided > one ticket, but the waiters have taken two. c.tickets goes > negative. > > After this, when some thread can do another (M3-)Wait, and some > other does another (M3-)Signal, c.tickets increments only back to > zero, which means the new waiter will not M3-wake, even though it > should. > > > I am going to try to construct a test case that will force > this scenario. > > See more below: > > > On 07/18/2016 08:53 PM, Jay wrote: > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > Otherwise previous version I think had a very large lock. > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > I guess you are saying it would be unfair, which is ok. > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > I'll check on the unused event. > > > > - Jay > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > >> > >> > >> > >> > >> > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > >> comments: > >> > >> This looks like a very direct implementation of Schmidts' generation count > >> algorithm, with a number of important little details additionally taken care > >> of. > >> > >> On small difference is at line 308 (B, below) where Schmidt does > >> c.counter>count instead of #. Usually, this would make no difference, > >> since counter only increases, but on a long-running program, it could > >> overflow. If this happened,as it often does, as silent wrap-around to > >> FIRST(INTEGER), some threads could be trapped for an extremely long > >> time, waiting for counter come back around greater than their count. As > >> coded in ThreadWin32, they would be allowed to proceed normally. > >> > >> The only glitch would be that, if there were simultaneously waiting threads > >> separated by NUMBER(INTEGER) truly different generations (but the same value > >> of c.counter), the recent ones could unfairly proceed along with the > >> extremely old ones. That seems extremely less likely than the > >> mere existence of an overflow. > >> > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > >> only created at :537 and deleted at :554, but never used in any real > >> way. It could be deleted. > >> > >> As for this code: > >> > >> EnterCriticalSection(conditionLock); > >> > >> (* Capture the value of the counter before we start waiting. > >> * We will not stop waiting until the counter changes. > >> * That is, we will not stop waiting until a signal > >> * comes in after we start waiting. > >> *) > >> > >> count := c.counter; > >> INC(c.waiters); > >> > >> LeaveCriticalSection(conditionLock); > >> m.release(); (* can this be moved to before taking conditionLock? *) > >> > >> No, it is important to sample and save c.counter in the order threads wait. > >> Retaining m until this is done ensures this. Otherwise, some other thread > >> that waited later could have altered c.counter before this one gets conditionLock > >> and saves its copy of c.counter. > >> > >> I have made several comment changes to ThreadWin32.m3 that would have made > >> it easier to vet. I will commit these, but only comments. It is either > >> very difficult or impossible for me to test or even recompile this module, > >> and, especially as fragile as this kind of code is, I don't want to commit > >> any substantive changes untested and uncompiled. > >> > >> More below: > >> > >>> On 07/07/2016 04:24 AM, Jay K wrote: > >>> So...I do NOT understand all of this. > >>> > >>> > >>> However, comparing the three implementations > >>> and attempting to understand ours, > >>> I am struck by the following > >>> > >>> - Our broadcast seems ok, but > >>> - our signal seems to wake everyone, > >>> and then...they race a bit, > >>> one will decide it is last, and > >>> reset the event, but every prior waiter > >>> is still woken. > >>> Why not merge the following chunks: > >> > >> I don't think this will work. It is important that, once a thread has decided, > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > >> if it is also the last of its generation to proceed, resets c.waitevent. This > >> is ensured by its having already reacquired m first, preventing any other thread > >> from returning from its Wait yet. > > I was wrong about this. There is already no fairness being enforced here. Even with > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > Merging, as you proposed, is probably close to the fix for this bug. > > > >> > >> And it can't attempt to acquire m while already holding conditionLock, because this would > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > >> already holding m, at line 265. > >> > >>> > >>> > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > >>> . > >>> . > >>> . > >>> 1 > >>> EnterCriticalSection(conditionLock); > >>> waitDone := (c.tickets # 0 AND c.counter # count); > >> A:---------^ > >> B:--------------------------------------------------^ > >>> LeaveCriticalSection(conditionLock); > >>> END; (* WHILE *) > >>> IF waitDone THEN > >>> alerted := FALSE; > >>> END; > >>> m.acquire(); > >>> 2 > >>> EnterCriticalSection(conditionLock); > >>> DEC(c.waiters); > >>> IF waitDone THEN esp. here. > >>> DEC(c.tickets); > >> > >> C:---------------^ > >>> lastWaiter := (c.tickets = 0); > >>> END; > >>> LeaveCriticalSection(conditionLock); > >>> > >>> > >>> That is, if we decrement tickets earlier > >>> within the first critical section, > >>> while we will still wake everyone, > >>> only one will decide waitDone and the rest will keep looping. > >>> A downside of this, perhaps, is that waking all for Broadcast > >>> might be a little slower. > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > >>> implementations, they seem to deal with this differently than us, > >>> and they each do about the same thing -- they use a counted semaphore. > >>> In the boost case, it appears they duplicate the semaphore for every > >>> notification generation, which seems expensive. > >>> > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > >>> Or they can do their own reference counting? > >>> jdk7 seems to looke like jdk6. > >>> The code is gone in jdk8 and I can't easily find the delete in history. > >>> The lock merging jdk does probably helps here too. > >>> In fact they merge #1 and #2 above as a result. > >>> But they still initially wake all threads for signal, not just broadcast. > >>> There is also the problem that all the event waits > >>> are followed by EnterCriticalSection (or jdk facsimile). > >>> - Jay > >>> > >>> > >>> > >>> > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! > >> --- > >>> From: jay.krell at cornell.edu > >>> To: m3devel at elegosoft.com > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > >>> > >>> Here is another implementation to consider: > >>> > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > >>> > >>> - Jay > >>> > >>> > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! > >> --- > >>> From: jay.krell at cornell.edu > >>> To: m3devel at elegosoft.com > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > >>> Subject: [M3devel] purported condition variable problems on Win32? > >>> > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > >>> > >>> vs. > >>> > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >>> vs. > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > >>> I wrote this: > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > >>> alertable: BOOLEAN) RAISES {Alerted} = > >>> (* LL = m on entry and exit, but not for the duration > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > >>> * NOTE that they merge the user lock and the condition lock. > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >>> * "3.3. The Generation Count Solution" > >>> *) > >>> > >>> Do we have the problems described in README.CV? > >>> > >>> I haven't looked through the ACE code to see > >>> to what extent they resemble solution 3.3, or if they > >>> changed as a result of this discussion -- which I admit I don't understand > >>> and haven't read closely. > >>> > >>> > >>> Spurious wakeups are ok, though should be minimized. > >>> > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > >>> > >>> Thank you, > >>> - Jay > >>> > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >>> > >>> > >>> _______________________________________________ > >>> M3devel mailing list > >>> M3devel at elegosoft.com > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >> > >> -- > >> Rodney Bates > >> rodney.m.bates at acm.org > >> > >> > > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Thu Jul 21 19:01:38 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Thu, 21 Jul 2016 17:01:38 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop> Message-ID: <5790FF55.5030007@lcwb.coop> On 07/21/2016 01:35 AM, Jay K wrote: > Am I understanding this? > There are some waiters. > There is a broadcast ("wake all"). Some waiters are woken. > There are more waiters. > There is a signal ("wake one"). > There is a chance that some of the original waiters will remain waiting while the later waiters are not. > That is, the signaled threads kind of steal the wake intended by the broadcast. > > > Is this unfairness or incorrectness? > It feels like unfairness. > It feels like condition variables are specified fairly weakly. > It's incorrectness. The older waiters were waiting at the time of the Broadcast and should all M3-awaken, but one does not. As a aside, more than one of the newer waiters M3-awakens in response to the Signal, which is more than necessary, but OK, the way M3 Signal is specified. It's all because the ticket mechanism only controls the number of treads to wake up, but, in this case, it matters not just how many but which one(s). > > - Jay > > > > Date: Wed, 20 Jul 2016 11:43:27 -0500 > > From: rodney_bates at lcwb.coop > > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] purported condition variable problems on Win32? > > > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > > > This gets confusing, since there are Modula-3 waits, signals, > > etc. and similar concepts in Win32, but which are not the > > same. I am going the try to put "M3-" and "Win-" prefixes on > > things to make this clearer. I cartainly need to do this > > for my own benefit, if for nobody else. > > > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > > c.tickets = 0, and c.waiters = 2. > > > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > > which will allow both the waiters to Win-wakeup sometime soon. > > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > > to eventually allow only one waiter to M3-wakeup) and increments > > c.counter, setting things up so that the waitDone expression at :308 > > will be TRUE for the waiting threads, the next time they evaluate it. > > > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > > releases c.lock, and tries to acquire m. The problem arises > > because the second waiter can get to these steps before the first > > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > > second to reach :308 will find c.tickets # 0, as did the first, > > and proceed. > > > > Each will eventually, do a M3-wake. If this were Posix condition > > variables, this would be wrong, because Posix says only one > > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > > says one or more waiters proceed. However, each will, in the > > process, get to :322, and DEC(c.tickets). Signal only provided > > one ticket, but the waiters have taken two. c.tickets goes > > negative. > > > > After this, when some thread can do another (M3-)Wait, and some > > other does another (M3-)Signal, c.tickets increments only back to > > zero, which means the new waiter will not M3-wake, even though it > > should. > > > > > > I am going to try to construct a test case that will force > > this scenario. > > > > See more below: > > > > > > On 07/18/2016 08:53 PM, Jay wrote: > > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > > > Otherwise previous version I think had a very large lock. > > > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > > > I guess you are saying it would be unfair, which is ok. > > > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > > > I'll check on the unused event. > > > > > > - Jay > > > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > > >> > > >> > > >> > > >> > > >> > > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > > >> comments: > > >> > > >> This looks like a very direct implementation of Schmidts' generation count > > >> algorithm, with a number of important little details additionally taken care > > >> of. > > >> > > >> On small difference is at line 308 (B, below) where Schmidt does > > >> c.counter>count instead of #. Usually, this would make no difference, > > >> since counter only increases, but on a long-running program, it could > > >> overflow. If this happened,as it often does, as silent wrap-around to > > >> FIRST(INTEGER), some threads could be trapped for an extremely long > > >> time, waiting for counter come back around greater than their count. As > > >> coded in ThreadWin32, they would be allowed to proceed normally. > > >> > > >> The only glitch would be that, if there were simultaneously waiting threads > > >> separated by NUMBER(INTEGER) truly different generations (but the same value > > >> of c.counter), the recent ones could unfairly proceed along with the > > >> extremely old ones. That seems extremely less likely than the > > >> mere existence of an overflow. > > >> > > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > > >> only created at :537 and deleted at :554, but never used in any real > > >> way. It could be deleted. > > >> > > >> As for this code: > > >> > > >> EnterCriticalSection(conditionLock); > > >> > > >> (* Capture the value of the counter before we start waiting. > > >> * We will not stop waiting until the counter changes. > > >> * That is, we will not stop waiting until a signal > > >> * comes in after we start waiting. > > >> *) > > >> > > >> count := c.counter; > > >> INC(c.waiters); > > >> > > >> LeaveCriticalSection(conditionLock); > > >> m.release(); (* can this be moved to before taking conditionLock? *) > > >> > > >> No, it is important to sample and save c.counter in the order threads wait. > > >> Retaining m until this is done ensures this. Otherwise, some other thread > > >> that waited later could have altered c.counter before this one gets conditionLock > > >> and saves its copy of c.counter. > > >> > > >> I have made several comment changes to ThreadWin32.m3 that would have made > > >> it easier to vet. I will commit these, but only comments. It is either > > >> very difficult or impossible for me to test or even recompile this module, > > >> and, especially as fragile as this kind of code is, I don't want to commit > > >> any substantive changes untested and uncompiled. > > >> > > >> More below: > > >> > > >>> On 07/07/2016 04:24 AM, Jay K wrote: > > >>> So...I do NOT understand all of this. > > >>> > > >>> > > >>> However, comparing the three implementations > > >>> and attempting to understand ours, > > >>> I am struck by the following > > >>> > > >>> - Our broadcast seems ok, but > > >>> - our signal seems to wake everyone, > > >>> and then...they race a bit, > > >>> one will decide it is last, and > > >>> reset the event, but every prior waiter > > >>> is still woken. > > >>> Why not merge the following chunks: > > >> > > >> I don't think this will work. It is important that, once a thread has decided, > > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > > >> if it is also the last of its generation to proceed, resets c.waitevent. This > > >> is ensured by its having already reacquired m first, preventing any other thread > > >> from returning from its Wait yet. > > > > I was wrong about this. There is already no fairness being enforced here. Even with > > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > > > Merging, as you proposed, is probably close to the fix for this bug. > > > > > > >> > > >> And it can't attempt to acquire m while already holding conditionLock, because this would > > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > > >> already holding m, at line 265. > > >> > > >>> > > >>> > > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > > >>> . > > >>> . > > >>> . > > >>> 1 > > >>> EnterCriticalSection(conditionLock); > > >>> waitDone := (c.tickets # 0 AND c.counter # count); > > >> A:---------^ > > >> B:--------------------------------------------------^ > > >>> LeaveCriticalSection(conditionLock); > > >>> END; (* WHILE *) > > >>> IF waitDone THEN > > >>> alerted := FALSE; > > >>> END; > > >>> m.acquire(); > > >>> 2 > > >>> EnterCriticalSection(conditionLock); > > >>> DEC(c.waiters); > > >>> IF waitDone THEN esp. here. > > >>> DEC(c.tickets); > > >> > > >> C:---------------^ > > >>> lastWaiter := (c.tickets = 0); > > >>> END; > > >>> LeaveCriticalSection(conditionLock); > > >>> > > >>> > > >>> That is, if we decrement tickets earlier > > >>> within the first critical section, > > >>> while we will still wake everyone, > > >>> only one will decide waitDone and the rest will keep looping. > > >>> A downside of this, perhaps, is that waking all for Broadcast > > >>> might be a little slower. > > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > > >>> implementations, they seem to deal with this differently than us, > > >>> and they each do about the same thing -- they use a counted semaphore. > > >>> In the boost case, it appears they duplicate the semaphore for every > > >>> notification generation, which seems expensive. > > >>> > > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > > >>> Or they can do their own reference counting? > > >>> jdk7 seems to looke like jdk6. > > >>> The code is gone in jdk8 and I can't easily find the delete in history. > > >>> The lock merging jdk does probably helps here too. > > >>> In fact they merge #1 and #2 above as a result. > > >>> But they still initially wake all threads for signal, not just broadcast. > > >>> There is also the problem that all the event waits > > >>> are followed by EnterCriticalSection (or jdk facsimile). > > >>> - Jay > > >>> > > >>> > > >>> > > >>> > > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > --! > > >> --- > > >>> From: jay.krell at cornell.edu > > >>> To: m3devel at elegosoft.com > > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > > >>> > > >>> Here is another implementation to consider: > > >>> > > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > > >>> > > >>> - Jay > > >>> > > >>> > > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > --! > > >> --- > > >>> From: jay.krell at cornell.edu > > >>> To: m3devel at elegosoft.com > > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > > >>> Subject: [M3devel] purported condition variable problems on Win32? > > >>> > > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > > >>> > > >>> vs. > > >>> > > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > >>> vs. > > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > > >>> I wrote this: > > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > > >>> alertable: BOOLEAN) RAISES {Alerted} = > > >>> (* LL = m on entry and exit, but not for the duration > > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > > >>> * NOTE that they merge the user lock and the condition lock. > > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > >>> * "3.3. The Generation Count Solution" > > >>> *) > > >>> > > >>> Do we have the problems described in README.CV? > > >>> > > >>> I haven't looked through the ACE code to see > > >>> to what extent they resemble solution 3.3, or if they > > >>> changed as a result of this discussion -- which I admit I don't understand > > >>> and haven't read closely. > > >>> > > >>> > > >>> Spurious wakeups are ok, though should be minimized. > > >>> > > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > >>> > > >>> Thank you, > > >>> - Jay > > >>> > > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > >>> > > >>> > > >>> _______________________________________________ > > >>> M3devel mailing list > > >>> M3devel at elegosoft.com > > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > >> > > >> -- > > >> Rodney Bates > > >> rodney.m.bates at acm.org > > >> > > >> > > > > > > > -- > > Rodney Bates > > rodney.m.bates at acm.org -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Thu Jul 21 19:04:40 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Thu, 21 Jul 2016 17:04:40 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop>, <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop>, Message-ID: <57910079.6090705@lcwb.coop> I think the first lost-wakeup bug can be fixed by merging the two lock sections on c.lock or maybe moving just the DEC(c.tickets) up to the first. I would need to write sample code, then review it to be confident. I have not done that so far. Maybe the Java folks knew this and got it right. On 07/21/2016 01:43 AM, Jay K wrote: > That question was meant for the other bug. > > Can this one be fixed by rechecking if tickets is positive before decrementing it? > This seems like a big problem in Schmidt's code. > > I'm nervous about merging the locks, but maybe. > In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. > Though I realize that is actually necessarily part of what gets done. > Schmidt clearly did not merge the locks, and the Java version does. > The Java version also has its own simple and not-terrible critical section. > The Java version also disappeared in newer releases and I didn't track down what happened to it. > > - Jay > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > From: jay.krell at cornell.edu > To: rodney.m.bates at acm.org > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] purported condition variable problems on Win32? > Date: Thu, 21 Jul 2016 06:35:41 +0000 > > Am I understanding this? > There are some waiters. > There is a broadcast ("wake all"). Some waiters are woken. > There are more waiters. > There is a signal ("wake one"). > There is a chance that some of the original waiters will remain waiting while the later waiters are not. > That is, the signaled threads kind of steal the wake intended by the broadcast. > > > Is this unfairness or incorrectness? > It feels like unfairness. > It feels like condition variables are specified fairly weakly. > > > - Jay > > > > Date: Wed, 20 Jul 2016 11:43:27 -0500 > > From: rodney_bates at lcwb.coop > > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] purported condition variable problems on Win32? > > > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > > > This gets confusing, since there are Modula-3 waits, signals, > > etc. and similar concepts in Win32, but which are not the > > same. I am going the try to put "M3-" and "Win-" prefixes on > > things to make this clearer. I cartainly need to do this > > for my own benefit, if for nobody else. > > > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > > c.tickets = 0, and c.waiters = 2. > > > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > > which will allow both the waiters to Win-wakeup sometime soon. > > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > > to eventually allow only one waiter to M3-wakeup) and increments > > c.counter, setting things up so that the waitDone expression at :308 > > will be TRUE for the waiting threads, the next time they evaluate it. > > > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > > releases c.lock, and tries to acquire m. The problem arises > > because the second waiter can get to these steps before the first > > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > > second to reach :308 will find c.tickets # 0, as did the first, > > and proceed. > > > > Each will eventually, do a M3-wake. If this were Posix condition > > variables, this would be wrong, because Posix says only one > > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > > says one or more waiters proceed. However, each will, in the > > process, get to :322, and DEC(c.tickets). Signal only provided > > one ticket, but the waiters have taken two. c.tickets goes > > negative. > > > > After this, when some thread can do another (M3-)Wait, and some > > other does another (M3-)Signal, c.tickets increments only back to > > zero, which means the new waiter will not M3-wake, even though it > > should. > > > > > > I am going to try to construct a test case that will force > > this scenario. > > > > See more below: > > > > > > On 07/18/2016 08:53 PM, Jay wrote: > > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > > > Otherwise previous version I think had a very large lock. > > > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > > > I guess you are saying it would be unfair, which is ok. > > > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > > > I'll check on the unused event. > > > > > > - Jay > > > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > > >> > > >> > > >> > > >> > > >> > > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > > >> comments: > > >> > > >> This looks like a very direct implementation of Schmidts' generation count > > >> algorithm, with a number of important little details additionally taken care > > >> of. > > >> > > >> On small difference is at line 308 (B, below) where Schmidt does > > >> c.counter>count instead of #. Usually, this would make no difference, > > >> since counter only increases, but on a long-running program, it could > > >> overflow. If this happened,as it often does, as silent wrap-around to > > >> FIRST(INTEGER), some threads could be trapped for an extremely long > > >> time, waiting for counter come back around greater than their count. As > > >> coded in ThreadWin32, they would be allowed to proceed normally. > > >> > > >> The only glitch would be that, if there were simultaneously waiting threads > > >> separated by NUMBER(INTEGER) truly different generations (but the same value > > >> of c.counter), the recent ones could unfairly proceed along with the > > >> extremely old ones. That seems extremely less likely than the > > >> mere existence of an overflow. > > >> > > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > > >> only created at :537 and deleted at :554, but never used in any real > > >> way. It could be deleted. > > >> > > >> As for this code: > > >> > > >> EnterCriticalSection(conditionLock); > > >> > > >> (* Capture the value of the counter before we start waiting. > > >> * We will not stop waiting until the counter changes. > > >> * That is, we will not stop waiting until a signal > > >> * comes in after we start waiting. > > >> *) > > >> > > >> count := c.counter; > > >> INC(c.waiters); > > >> > > >> LeaveCriticalSection(conditionLock); > > >> m.release(); (* can this be moved to before taking conditionLock? *) > > >> > > >> No, it is important to sample and save c.counter in the order threads wait. > > >> Retaining m until this is done ensures this. Otherwise, some other thread > > >> that waited later could have altered c.counter before this one gets conditionLock > > >> and saves its copy of c.counter. > > >> > > >> I have made several comment changes to ThreadWin32.m3 that would have made > > >> it easier to vet. I will commit these, but only comments. It is either > > >> very difficult or impossible for me to test or even recompile this module, > > >> and, especially as fragile as this kind of code is, I don't want to commit > > >> any substantive changes untested and uncompiled. > > >> > > >> More below: > > >> > > >>> On 07/07/2016 04:24 AM, Jay K wrote: > > >>> So...I do NOT understand all of this. > > >>> > > >>> > > >>> However, comparing the three implementations > > >>> and attempting to understand ours, > > >>> I am struck by the following > > >>> > > >>> - Our broadcast seems ok, but > > >>> - our signal seems to wake everyone, > > >>> and then...they race a bit, > > >>> one will decide it is last, and > > >>> reset the event, but every prior waiter > > >>> is still woken. > > >>> Why not merge the following chunks: > > >> > > >> I don't think this will work. It is important that, once a thread has decided, > > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > > >> if it is also the last of its generation to proceed, resets c.waitevent. This > > >> is ensured by its having already reacquired m first, preventing any other thread > > >> from returning from its Wait yet. > > > > I was wrong about this. There is already no fairness being enforced here. Even with > > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > > > Merging, as you proposed, is probably close to the fix for this bug. > > > > > > >> > > >> And it can't attempt to acquire m while already holding conditionLock, because this would > > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > > >> already holding m, at line 265. > > >> > > >>> > > >>> > > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > > >>> . > > >>> . > > >>> . > > >>> 1 > > >>> EnterCriticalSection(conditionLock); > > >>> waitDone := (c.tickets # 0 AND c.counter # count); > > >> A:---------^ > > >> B:--------------------------------------------------^ > > >>> LeaveCriticalSection(conditionLock); > > >>> END; (* WHILE *) > > >>> IF waitDone THEN > > >>> alerted := FALSE; > > >>> END; > > >>> m.acquire(); > > >>> 2 > > >>> EnterCriticalSection(conditionLock); > > >>> DEC(c.waiters); > > >>> IF waitDone THEN esp. here. > > >>> DEC(c.tickets); > > >> > > >> C:---------------^ > > >>> lastWaiter := (c.tickets = 0); > > >>> END; > > >>> LeaveCriticalSection(conditionLock); > > >>> > > >>> > > >>> That is, if we decrement tickets earlier > > >>> within the first critical section, > > >>> while we will still wake everyone, > > >>> only one will decide waitDone and the rest will keep looping. > > >>> A downside of this, perhaps, is that waking all for Broadcast > > >>> might be a little slower. > > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > > >>> implementations, they seem to deal with this differently than us, > > >>> and they each do about the same thing -- they use a counted semaphore. > > >>> In the boost case, it appears they duplicate the semaphore for every > > >>> notification generation, which seems expensive. > > >>> > > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > > >>> Or they can do their own reference counting? > > >>> jdk7 seems to looke like jdk6. > > >>> The code is gone in jdk8 and I can't easily find the delete in history. > > >>> The lock merging jdk does probably helps here too. > > >>> In fact they merge #1 and #2 above as a result. > > >>> But they still initially wake all threads for signal, not just broadcast. > > >>> There is also the problem that all the event waits > > >>> are followed by EnterCriticalSection (or jdk facsimile). > > >>> - Jay > > >>> > > >>> > > >>> > > >>> > > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > --! > > >> --- > > >>> From: jay.krell at cornell.edu > > >>> To: m3devel at elegosoft.com > > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > > >>> > > >>> Here is another implementation to consider: > > >>> > > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > > >>> > > >>> - Jay > > >>> > > >>> > > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > --! > > >> --- > > >>> From: jay.krell at cornell.edu > > >>> To: m3devel at elegosoft.com > > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > > >>> Subject: [M3devel] purported condition variable problems on Win32? > > >>> > > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > > >>> > > >>> vs. > > >>> > > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > >>> vs. > > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > > >>> I wrote this: > > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > > >>> alertable: BOOLEAN) RAISES {Alerted} = > > >>> (* LL = m on entry and exit, but not for the duration > > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > > >>> * NOTE that they merge the user lock and the condition lock. > > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > >>> * "3.3. The Generation Count Solution" > > >>> *) > > >>> > > >>> Do we have the problems described in README.CV? > > >>> > > >>> I haven't looked through the ACE code to see > > >>> to what extent they resemble solution 3.3, or if they > > >>> changed as a result of this discussion -- which I admit I don't understand > > >>> and haven't read closely. > > >>> > > >>> > > >>> Spurious wakeups are ok, though should be minimized. > > >>> > > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > >>> > > >>> Thank you, > > >>> - Jay > > >>> > > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > >>> > > >>> > > >>> _______________________________________________ > > >>> M3devel mailing list > > >>> M3devel at elegosoft.com > > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > >> > > >> -- > > >> Rodney Bates > > >> rodney.m.bates at acm.org > > >> > > >> > > > > > > > -- > > Rodney Bates > > rodney.m.bates at acm.org -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Fri Jul 22 06:24:45 2016 From: jay.krell at cornell.edu (Jay) Date: Fri, 22 Jul 2016 04:24:45 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: <57910079.6090705@lcwb.coop> References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop> <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> <578FAA2F.6020105@lcwb.coop> <57910079.6090705@lcwb.coop> Message-ID: <0C7BFA25-E353-4E89-A739-C33CF5290A6E@gmail.com> Do our semantics allow merging the locks? I.e. Must always use same lock with same condition and vice versa? That would indeed be a nice optimization, if the interface is limited that way. I'll look around later... - Jay > On Jul 21, 2016, at 10:03 AM, "Rodney M. Bates" wrote: > > I think the first lost-wakeup bug can be fixed by merging the two lock sections > on c.lock or maybe moving just the DEC(c.tickets) up to the first. I would > need to write sample code, then review it to be confident. I have not done > that so far. > > Maybe the Java folks knew this and got it right. > >> On 07/21/2016 01:43 AM, Jay K wrote: >> That question was meant for the other bug. >> >> Can this one be fixed by rechecking if tickets is positive before decrementing it? >> This seems like a big problem in Schmidt's code. >> >> I'm nervous about merging the locks, but maybe. >> In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. >> Though I realize that is actually necessarily part of what gets done. >> Schmidt clearly did not merge the locks, and the Java version does. >> The Java version also has its own simple and not-terrible critical section. >> The Java version also disappeared in newer releases and I didn't track down what happened to it. >> >> - Jay >> >> >> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --- >> From: jay.krell at cornell.edu >> To: rodney.m.bates at acm.org >> CC: m3devel at elegosoft.com >> Subject: RE: [M3devel] purported condition variable problems on Win32? >> Date: Thu, 21 Jul 2016 06:35:41 +0000 >> >> Am I understanding this? >> There are some waiters. >> There is a broadcast ("wake all"). Some waiters are woken. >> There are more waiters. >> There is a signal ("wake one"). >> There is a chance that some of the original waiters will remain waiting while the later waiters are not. >> That is, the signaled threads kind of steal the wake intended by the broadcast. >> >> >> Is this unfairness or incorrectness? >> It feels like unfairness. >> It feels like condition variables are specified fairly weakly. >> >> >> - Jay >> >> >> > Date: Wed, 20 Jul 2016 11:43:27 -0500 >> > From: rodney_bates at lcwb.coop >> > To: jay.krell at cornell.edu; rodney.m.bates at acm.org >> > CC: m3devel at elegosoft.com >> > Subject: Re: [M3devel] purported condition variable problems on Win32? >> > >> > In looking at ThreadWin32.m3 more, I think I see a significant bug. >> > >> > This gets confusing, since there are Modula-3 waits, signals, >> > etc. and similar concepts in Win32, but which are not the >> > same. I am going the try to put "M3-" and "Win-" prefixes on >> > things to make this clearer. I cartainly need to do this >> > for my own benefit, if for nobody else. >> > >> > Suppose two threads are both M3-waiting in Wait(m,c,...), further >> > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. >> > c.tickets = 0, and c.waiters = 2. >> > >> > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), >> > which will allow both the waiters to Win-wakeup sometime soon. >> > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly >> > to eventually allow only one waiter to M3-wakeup) and increments >> > c.counter, setting things up so that the waitDone expression at :308 >> > will be TRUE for the waiting threads, the next time they evaluate it. >> > >> > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, >> > releases c.lock, and tries to acquire m. The problem arises >> > because the second waiter can get to these steps before the first >> > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The >> > second to reach :308 will find c.tickets # 0, as did the first, >> > and proceed. >> > >> > Each will eventually, do a M3-wake. If this were Posix condition >> > variables, this would be wrong, because Posix says only one >> > waiter proceeds. For M3, it's OK, if unnecessary, because M3 >> > says one or more waiters proceed. However, each will, in the >> > process, get to :322, and DEC(c.tickets). Signal only provided >> > one ticket, but the waiters have taken two. c.tickets goes >> > negative. >> > >> > After this, when some thread can do another (M3-)Wait, and some >> > other does another (M3-)Signal, c.tickets increments only back to >> > zero, which means the new waiter will not M3-wake, even though it >> > should. >> > >> > >> > I am going to try to construct a test case that will force >> > this scenario. >> > >> > See more below: >> > >> > >> > On 07/18/2016 08:53 PM, Jay wrote: >> > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. >> > > >> > > Otherwise previous version I think had a very large lock. >> > > >> > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. >> > > >> > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. >> > > >> > > I guess you are saying it would be unfair, which is ok. >> > > >> > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. >> > > >> > > I'll check on the unused event. >> > > >> > > - Jay >> > > >> > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> I spent some time looking over the signal part of ThreadWin32.m3. A few >> > >> comments: >> > >> >> > >> This looks like a very direct implementation of Schmidts' generation count >> > >> algorithm, with a number of important little details additionally taken care >> > >> of. >> > >> >> > >> On small difference is at line 308 (B, below) where Schmidt does >> > >> c.counter>count instead of #. Usually, this would make no difference, >> > >> since counter only increases, but on a long-running program, it could >> > >> overflow. If this happened,as it often does, as silent wrap-around to >> > >> FIRST(INTEGER), some threads could be trapped for an extremely long >> > >> time, waiting for counter come back around greater than their count. As >> > >> coded in ThreadWin32, they would be allowed to proceed normally. >> > >> >> > >> The only glitch would be that, if there were simultaneously waiting threads >> > >> separated by NUMBER(INTEGER) truly different generations (but the same value >> > >> of c.counter), the recent ones could unfairly proceed along with the >> > >> extremely old ones. That seems extremely less likely than the >> > >> mere existence of an overflow. >> > >> >> > >> Both type Condition and type Activation have a field waitEvent: HANDLE. >> > >> Condition.waitEvent is extensively used, but Activation.waitEvent is >> > >> only created at :537 and deleted at :554, but never used in any real >> > >> way. It could be deleted. >> > >> >> > >> As for this code: >> > >> >> > >> EnterCriticalSection(conditionLock); >> > >> >> > >> (* Capture the value of the counter before we start waiting. >> > >> * We will not stop waiting until the counter changes. >> > >> * That is, we will not stop waiting until a signal >> > >> * comes in after we start waiting. >> > >> *) >> > >> >> > >> count := c.counter; >> > >> INC(c.waiters); >> > >> >> > >> LeaveCriticalSection(conditionLock); >> > >> m.release(); (* can this be moved to before taking conditionLock? *) >> > >> >> > >> No, it is important to sample and save c.counter in the order threads wait. >> > >> Retaining m until this is done ensures this. Otherwise, some other thread >> > >> that waited later could have altered c.counter before this one gets conditionLock >> > >> and saves its copy of c.counter. >> > >> >> > >> I have made several comment changes to ThreadWin32.m3 that would have made >> > >> it easier to vet. I will commit these, but only comments. It is either >> > >> very difficult or impossible for me to test or even recompile this module, >> > >> and, especially as fragile as this kind of code is, I don't want to commit >> > >> any substantive changes untested and uncompiled. >> > >> >> > >> More below: >> > >> >> > >>> On 07/07/2016 04:24 AM, Jay K wrote: >> > >>> So...I do NOT understand all of this. >> > >>> >> > >>> >> > >>> However, comparing the three implementations >> > >>> and attempting to understand ours, >> > >>> I am struck by the following >> > >>> >> > >>> - Our broadcast seems ok, but >> > >>> - our signal seems to wake everyone, >> > >>> and then...they race a bit, >> > >>> one will decide it is last, and >> > >>> reset the event, but every prior waiter >> > >>> is still woken. >> > >>> Why not merge the following chunks: >> > >> >> > >> I don't think this will work. It is important that, once a thread has decided, >> > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed >> > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, >> > >> if it is also the last of its generation to proceed, resets c.waitevent. This >> > >> is ensured by its having already reacquired m first, preventing any other thread >> > >> from returning from its Wait yet. >> > >> > I was wrong about this. There is already no fairness being enforced here. Even with >> > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is >> > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. >> > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters >> > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. >> > >> > Merging, as you proposed, is probably close to the fix for this bug. >> > >> > >> > >> >> > >> And it can't attempt to acquire m while already holding conditionLock, because this would >> > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while >> > >> already holding m, at line 265. >> > >> >> > >>> >> > >>> >> > >>> WHILE (NOT alerted) AND (NOT waitDone) DO >> > >>> . >> > >>> . >> > >>> . >> > >>> 1 >> > >>> EnterCriticalSection(conditionLock); >> > >>> waitDone := (c.tickets # 0 AND c.counter # count); >> > >> A:---------^ >> > >> B:--------------------------------------------------^ >> > >>> LeaveCriticalSection(conditionLock); >> > >>> END; (* WHILE *) >> > >>> IF waitDone THEN >> > >>> alerted := FALSE; >> > >>> END; >> > >>> m.acquire(); >> > >>> 2 >> > >>> EnterCriticalSection(conditionLock); >> > >>> DEC(c.waiters); >> > >>> IF waitDone THEN esp. here. >> > >>> DEC(c.tickets); >> > >> >> > >> C:---------------^ >> > >>> lastWaiter := (c.tickets = 0); >> > >>> END; >> > >>> LeaveCriticalSection(conditionLock); >> > >>> >> > >>> >> > >>> That is, if we decrement tickets earlier >> > >>> within the first critical section, >> > >>> while we will still wake everyone, >> > >>> only one will decide waitDone and the rest will keep looping. >> > >>> A downside of this, perhaps, is that waking all for Broadcast >> > >>> might be a little slower. >> > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads >> > >>> implementations, they seem to deal with this differently than us, >> > >>> and they each do about the same thing -- they use a counted semaphore. >> > >>> In the boost case, it appears they duplicate the semaphore for every >> > >>> notification generation, which seems expensive. >> > >>> >> > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. >> > >>> Or they can do their own reference counting? >> > >>> jdk7 seems to looke like jdk6. >> > >>> The code is gone in jdk8 and I can't easily find the delete in history. >> > >>> The lock merging jdk does probably helps here too. >> > >>> In fact they merge #1 and #2 above as a result. >> > >>> But they still initially wake all threads for signal, not just broadcast. >> > >>> There is also the problem that all the event waits >> > >>> are followed by EnterCriticalSection (or jdk facsimile). >> > >>> - Jay >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > -----! >> > --! >> > >> --- >> > >>> From: jay.krell at cornell.edu >> > >>> To: m3devel at elegosoft.com >> > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? >> > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 >> > >>> >> > >>> Here is another implementation to consider: >> > >>> >> > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp >> > >>> >> > >>> - Jay >> > >>> >> > >>> >> > >>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > -----! >> > --! >> > >> --- >> > >>> From: jay.krell at cornell.edu >> > >>> To: m3devel at elegosoft.com >> > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 >> > >>> Subject: [M3devel] purported condition variable problems on Win32? >> > >>> >> > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV >> > >>> >> > >>> vs. >> > >>> >> > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >> > >>> vs. >> > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain >> > >>> I wrote this: >> > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; >> > >>> alertable: BOOLEAN) RAISES {Alerted} = >> > >>> (* LL = m on entry and exit, but not for the duration >> > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp >> > >>> * NOTE that they merge the user lock and the condition lock. >> > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >> > >>> * "3.3. The Generation Count Solution" >> > >>> *) >> > >>> >> > >>> Do we have the problems described in README.CV? >> > >>> >> > >>> I haven't looked through the ACE code to see >> > >>> to what extent they resemble solution 3.3, or if they >> > >>> changed as a result of this discussion -- which I admit I don't understand >> > >>> and haven't read closely. >> > >>> >> > >>> >> > >>> Spurious wakeups are ok, though should be minimized. >> > >>> >> > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. >> > >>> >> > >>> Thank you, >> > >>> - Jay >> > >>> >> > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> > >>> >> > >>> >> > >>> _______________________________________________ >> > >>> M3devel mailing list >> > >>> M3devel at elegosoft.com >> > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> > >> >> > >> -- >> > >> Rodney Bates >> > >> rodney.m.bates at acm.org >> > >> >> > >> >> > > >> > >> > -- >> > Rodney Bates >> > rodney.m.bates at acm.org > > -- > Rodney Bates > rodney.m.bates at acm.org From jay.krell at cornell.edu Fri Jul 22 10:20:36 2016 From: jay.krell at cornell.edu (Jay K) Date: Fri, 22 Jul 2016 08:20:36 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop>, <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop>, , Message-ID: Here is very weak circumstantial evidence that the locks can be merged: https://msdn.microsoft.com/en-us/library/windows/desktop/ms682052(v=vs.85).aspx "It is often convenient to use more than one condition variable with the same lock.For example, an implementation of a reader/writer lock might use a single criticalsection but separate condition variables for readers and writers." i.e. the mapping isn't one lock to one condition, but the convenientconstruct is multiple conditions to one lock, not multiple locks to one condition. http://linux.die.net/man/3/pthread_cond_wait "The effect of using more than one mutex for concurrent pthread_cond_timedwait() orpthread_cond_wait() operations on the same condition variable is undefined; that is,a condition variable becomes bound to a unique mutex when a thread waits on the conditionvariable, and this (dynamic) binding shall end when the wait returns." i.e. while the lock used with a condition variable can change through time,there can be only one at a time. And then, presumably win32 condition variables and posix condition variablesand Modula-3 Thread.Condition are all kind of similar. Hm, ok, a problem with merging the locks, is it leaves no lock for Signal/Broadcast.Perhaps they should use atomic/Interlocked. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org CC: m3devel at elegosoft.com Subject: RE: [M3devel] purported condition variable problems on Win32? Date: Thu, 21 Jul 2016 06:43:16 +0000 That question was meant for the other bug. Can this one be fixed by rechecking if tickets is positive before decrementing it? This seems like a big problem in Schmidt's code. I'm nervous about merging the locks, but maybe. In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. Though I realize that is actually necessarily part of what gets done. Schmidt clearly did not merge the locks, and the Java version does. The Java version also has its own simple and not-terrible critical section. The Java version also disappeared in newer releases and I didn't track down what happened to it. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org CC: m3devel at elegosoft.com Subject: RE: [M3devel] purported condition variable problems on Win32? Date: Thu, 21 Jul 2016 06:35:41 +0000 Am I understanding this? There are some waiters. There is a broadcast ("wake all"). Some waiters are woken. There are more waiters. There is a signal ("wake one"). There is a chance that some of the original waiters will remain waiting while the later waiters are not. That is, the signaled threads kind of steal the wake intended by the broadcast. Is this unfairness or incorrectness? It feels like unfairness. It feels like condition variables are specified fairly weakly. - Jay > Date: Wed, 20 Jul 2016 11:43:27 -0500 > From: rodney_bates at lcwb.coop > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] purported condition variable problems on Win32? > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > This gets confusing, since there are Modula-3 waits, signals, > etc. and similar concepts in Win32, but which are not the > same. I am going the try to put "M3-" and "Win-" prefixes on > things to make this clearer. I cartainly need to do this > for my own benefit, if for nobody else. > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > c.tickets = 0, and c.waiters = 2. > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > which will allow both the waiters to Win-wakeup sometime soon. > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > to eventually allow only one waiter to M3-wakeup) and increments > c.counter, setting things up so that the waitDone expression at :308 > will be TRUE for the waiting threads, the next time they evaluate it. > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > releases c.lock, and tries to acquire m. The problem arises > because the second waiter can get to these steps before the first > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > second to reach :308 will find c.tickets # 0, as did the first, > and proceed. > > Each will eventually, do a M3-wake. If this were Posix condition > variables, this would be wrong, because Posix says only one > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > says one or more waiters proceed. However, each will, in the > process, get to :322, and DEC(c.tickets). Signal only provided > one ticket, but the waiters have taken two. c.tickets goes > negative. > > After this, when some thread can do another (M3-)Wait, and some > other does another (M3-)Signal, c.tickets increments only back to > zero, which means the new waiter will not M3-wake, even though it > should. > > > I am going to try to construct a test case that will force > this scenario. > > See more below: > > > On 07/18/2016 08:53 PM, Jay wrote: > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > Otherwise previous version I think had a very large lock. > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > I guess you are saying it would be unfair, which is ok. > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > I'll check on the unused event. > > > > - Jay > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > >> > >> > >> > >> > >> > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > >> comments: > >> > >> This looks like a very direct implementation of Schmidts' generation count > >> algorithm, with a number of important little details additionally taken care > >> of. > >> > >> On small difference is at line 308 (B, below) where Schmidt does > >> c.counter>count instead of #. Usually, this would make no difference, > >> since counter only increases, but on a long-running program, it could > >> overflow. If this happened,as it often does, as silent wrap-around to > >> FIRST(INTEGER), some threads could be trapped for an extremely long > >> time, waiting for counter come back around greater than their count. As > >> coded in ThreadWin32, they would be allowed to proceed normally. > >> > >> The only glitch would be that, if there were simultaneously waiting threads > >> separated by NUMBER(INTEGER) truly different generations (but the same value > >> of c.counter), the recent ones could unfairly proceed along with the > >> extremely old ones. That seems extremely less likely than the > >> mere existence of an overflow. > >> > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > >> only created at :537 and deleted at :554, but never used in any real > >> way. It could be deleted. > >> > >> As for this code: > >> > >> EnterCriticalSection(conditionLock); > >> > >> (* Capture the value of the counter before we start waiting. > >> * We will not stop waiting until the counter changes. > >> * That is, we will not stop waiting until a signal > >> * comes in after we start waiting. > >> *) > >> > >> count := c.counter; > >> INC(c.waiters); > >> > >> LeaveCriticalSection(conditionLock); > >> m.release(); (* can this be moved to before taking conditionLock? *) > >> > >> No, it is important to sample and save c.counter in the order threads wait. > >> Retaining m until this is done ensures this. Otherwise, some other thread > >> that waited later could have altered c.counter before this one gets conditionLock > >> and saves its copy of c.counter. > >> > >> I have made several comment changes to ThreadWin32.m3 that would have made > >> it easier to vet. I will commit these, but only comments. It is either > >> very difficult or impossible for me to test or even recompile this module, > >> and, especially as fragile as this kind of code is, I don't want to commit > >> any substantive changes untested and uncompiled. > >> > >> More below: > >> > >>> On 07/07/2016 04:24 AM, Jay K wrote: > >>> So...I do NOT understand all of this. > >>> > >>> > >>> However, comparing the three implementations > >>> and attempting to understand ours, > >>> I am struck by the following > >>> > >>> - Our broadcast seems ok, but > >>> - our signal seems to wake everyone, > >>> and then...they race a bit, > >>> one will decide it is last, and > >>> reset the event, but every prior waiter > >>> is still woken. > >>> Why not merge the following chunks: > >> > >> I don't think this will work. It is important that, once a thread has decided, > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > >> if it is also the last of its generation to proceed, resets c.waitevent. This > >> is ensured by its having already reacquired m first, preventing any other thread > >> from returning from its Wait yet. > > I was wrong about this. There is already no fairness being enforced here. Even with > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > Merging, as you proposed, is probably close to the fix for this bug. > > > >> > >> And it can't attempt to acquire m while already holding conditionLock, because this would > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > >> already holding m, at line 265. > >> > >>> > >>> > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > >>> . > >>> . > >>> . > >>> 1 > >>> EnterCriticalSection(conditionLock); > >>> waitDone := (c.tickets # 0 AND c.counter # count); > >> A:---------^ > >> B:--------------------------------------------------^ > >>> LeaveCriticalSection(conditionLock); > >>> END; (* WHILE *) > >>> IF waitDone THEN > >>> alerted := FALSE; > >>> END; > >>> m.acquire(); > >>> 2 > >>> EnterCriticalSection(conditionLock); > >>> DEC(c.waiters); > >>> IF waitDone THEN esp. here. > >>> DEC(c.tickets); > >> > >> C:---------------^ > >>> lastWaiter := (c.tickets = 0); > >>> END; > >>> LeaveCriticalSection(conditionLock); > >>> > >>> > >>> That is, if we decrement tickets earlier > >>> within the first critical section, > >>> while we will still wake everyone, > >>> only one will decide waitDone and the rest will keep looping. > >>> A downside of this, perhaps, is that waking all for Broadcast > >>> might be a little slower. > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > >>> implementations, they seem to deal with this differently than us, > >>> and they each do about the same thing -- they use a counted semaphore. > >>> In the boost case, it appears they duplicate the semaphore for every > >>> notification generation, which seems expensive. > >>> > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > >>> Or they can do their own reference counting? > >>> jdk7 seems to looke like jdk6. > >>> The code is gone in jdk8 and I can't easily find the delete in history. > >>> The lock merging jdk does probably helps here too. > >>> In fact they merge #1 and #2 above as a result. > >>> But they still initially wake all threads for signal, not just broadcast. > >>> There is also the problem that all the event waits > >>> are followed by EnterCriticalSection (or jdk facsimile). > >>> - Jay > >>> > >>> > >>> > >>> > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! > >> --- > >>> From: jay.krell at cornell.edu > >>> To: m3devel at elegosoft.com > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > >>> > >>> Here is another implementation to consider: > >>> > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > >>> > >>> - Jay > >>> > >>> > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! > >> --- > >>> From: jay.krell at cornell.edu > >>> To: m3devel at elegosoft.com > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > >>> Subject: [M3devel] purported condition variable problems on Win32? > >>> > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > >>> > >>> vs. > >>> > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >>> vs. > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > >>> I wrote this: > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > >>> alertable: BOOLEAN) RAISES {Alerted} = > >>> (* LL = m on entry and exit, but not for the duration > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > >>> * NOTE that they merge the user lock and the condition lock. > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >>> * "3.3. The Generation Count Solution" > >>> *) > >>> > >>> Do we have the problems described in README.CV? > >>> > >>> I haven't looked through the ACE code to see > >>> to what extent they resemble solution 3.3, or if they > >>> changed as a result of this discussion -- which I admit I don't understand > >>> and haven't read closely. > >>> > >>> > >>> Spurious wakeups are ok, though should be minimized. > >>> > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > >>> > >>> Thank you, > >>> - Jay > >>> > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >>> > >>> > >>> _______________________________________________ > >>> M3devel mailing list > >>> M3devel at elegosoft.com > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >> > >> -- > >> Rodney Bates > >> rodney.m.bates at acm.org > >> > >> > > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Jul 22 10:32:27 2016 From: jay.krell at cornell.edu (Jay K) Date: Fri, 22 Jul 2016 08:32:27 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop>, <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop>, , , Message-ID: It works in Java because: - "mutex" and "condition" are merged into "monitor" - callers of notify/notifyAll are required to be holding the monitor's lock - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org CC: m3devel at elegosoft.com Subject: RE: [M3devel] purported condition variable problems on Win32? Date: Fri, 22 Jul 2016 08:20:32 +0000 Here is very weak circumstantial evidence that the locks can be merged: https://msdn.microsoft.com/en-us/library/windows/desktop/ms682052(v=vs.85).aspx "It is often convenient to use more than one condition variable with the same lock.For example, an implementation of a reader/writer lock might use a single criticalsection but separate condition variables for readers and writers." i.e. the mapping isn't one lock to one condition, but the convenientconstruct is multiple conditions to one lock, not multiple locks to one condition. http://linux.die.net/man/3/pthread_cond_wait "The effect of using more than one mutex for concurrent pthread_cond_timedwait() orpthread_cond_wait() operations on the same condition variable is undefined; that is,a condition variable becomes bound to a unique mutex when a thread waits on the conditionvariable, and this (dynamic) binding shall end when the wait returns." i.e. while the lock used with a condition variable can change through time,there can be only one at a time. And then, presumably win32 condition variables and posix condition variablesand Modula-3 Thread.Condition are all kind of similar. Hm, ok, a problem with merging the locks, is it leaves no lock for Signal/Broadcast.Perhaps they should use atomic/Interlocked. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org CC: m3devel at elegosoft.com Subject: RE: [M3devel] purported condition variable problems on Win32? Date: Thu, 21 Jul 2016 06:43:16 +0000 That question was meant for the other bug. Can this one be fixed by rechecking if tickets is positive before decrementing it? This seems like a big problem in Schmidt's code. I'm nervous about merging the locks, but maybe. In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. Though I realize that is actually necessarily part of what gets done. Schmidt clearly did not merge the locks, and the Java version does. The Java version also has its own simple and not-terrible critical section. The Java version also disappeared in newer releases and I didn't track down what happened to it. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org CC: m3devel at elegosoft.com Subject: RE: [M3devel] purported condition variable problems on Win32? Date: Thu, 21 Jul 2016 06:35:41 +0000 Am I understanding this? There are some waiters. There is a broadcast ("wake all"). Some waiters are woken. There are more waiters. There is a signal ("wake one"). There is a chance that some of the original waiters will remain waiting while the later waiters are not. That is, the signaled threads kind of steal the wake intended by the broadcast. Is this unfairness or incorrectness? It feels like unfairness. It feels like condition variables are specified fairly weakly. - Jay > Date: Wed, 20 Jul 2016 11:43:27 -0500 > From: rodney_bates at lcwb.coop > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] purported condition variable problems on Win32? > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > This gets confusing, since there are Modula-3 waits, signals, > etc. and similar concepts in Win32, but which are not the > same. I am going the try to put "M3-" and "Win-" prefixes on > things to make this clearer. I cartainly need to do this > for my own benefit, if for nobody else. > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > c.tickets = 0, and c.waiters = 2. > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > which will allow both the waiters to Win-wakeup sometime soon. > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > to eventually allow only one waiter to M3-wakeup) and increments > c.counter, setting things up so that the waitDone expression at :308 > will be TRUE for the waiting threads, the next time they evaluate it. > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > releases c.lock, and tries to acquire m. The problem arises > because the second waiter can get to these steps before the first > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > second to reach :308 will find c.tickets # 0, as did the first, > and proceed. > > Each will eventually, do a M3-wake. If this were Posix condition > variables, this would be wrong, because Posix says only one > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > says one or more waiters proceed. However, each will, in the > process, get to :322, and DEC(c.tickets). Signal only provided > one ticket, but the waiters have taken two. c.tickets goes > negative. > > After this, when some thread can do another (M3-)Wait, and some > other does another (M3-)Signal, c.tickets increments only back to > zero, which means the new waiter will not M3-wake, even though it > should. > > > I am going to try to construct a test case that will force > this scenario. > > See more below: > > > On 07/18/2016 08:53 PM, Jay wrote: > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > Otherwise previous version I think had a very large lock. > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > I guess you are saying it would be unfair, which is ok. > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > I'll check on the unused event. > > > > - Jay > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > >> > >> > >> > >> > >> > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > >> comments: > >> > >> This looks like a very direct implementation of Schmidts' generation count > >> algorithm, with a number of important little details additionally taken care > >> of. > >> > >> On small difference is at line 308 (B, below) where Schmidt does > >> c.counter>count instead of #. Usually, this would make no difference, > >> since counter only increases, but on a long-running program, it could > >> overflow. If this happened,as it often does, as silent wrap-around to > >> FIRST(INTEGER), some threads could be trapped for an extremely long > >> time, waiting for counter come back around greater than their count. As > >> coded in ThreadWin32, they would be allowed to proceed normally. > >> > >> The only glitch would be that, if there were simultaneously waiting threads > >> separated by NUMBER(INTEGER) truly different generations (but the same value > >> of c.counter), the recent ones could unfairly proceed along with the > >> extremely old ones. That seems extremely less likely than the > >> mere existence of an overflow. > >> > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > >> only created at :537 and deleted at :554, but never used in any real > >> way. It could be deleted. > >> > >> As for this code: > >> > >> EnterCriticalSection(conditionLock); > >> > >> (* Capture the value of the counter before we start waiting. > >> * We will not stop waiting until the counter changes. > >> * That is, we will not stop waiting until a signal > >> * comes in after we start waiting. > >> *) > >> > >> count := c.counter; > >> INC(c.waiters); > >> > >> LeaveCriticalSection(conditionLock); > >> m.release(); (* can this be moved to before taking conditionLock? *) > >> > >> No, it is important to sample and save c.counter in the order threads wait. > >> Retaining m until this is done ensures this. Otherwise, some other thread > >> that waited later could have altered c.counter before this one gets conditionLock > >> and saves its copy of c.counter. > >> > >> I have made several comment changes to ThreadWin32.m3 that would have made > >> it easier to vet. I will commit these, but only comments. It is either > >> very difficult or impossible for me to test or even recompile this module, > >> and, especially as fragile as this kind of code is, I don't want to commit > >> any substantive changes untested and uncompiled. > >> > >> More below: > >> > >>> On 07/07/2016 04:24 AM, Jay K wrote: > >>> So...I do NOT understand all of this. > >>> > >>> > >>> However, comparing the three implementations > >>> and attempting to understand ours, > >>> I am struck by the following > >>> > >>> - Our broadcast seems ok, but > >>> - our signal seems to wake everyone, > >>> and then...they race a bit, > >>> one will decide it is last, and > >>> reset the event, but every prior waiter > >>> is still woken. > >>> Why not merge the following chunks: > >> > >> I don't think this will work. It is important that, once a thread has decided, > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > >> if it is also the last of its generation to proceed, resets c.waitevent. This > >> is ensured by its having already reacquired m first, preventing any other thread > >> from returning from its Wait yet. > > I was wrong about this. There is already no fairness being enforced here. Even with > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > Merging, as you proposed, is probably close to the fix for this bug. > > > >> > >> And it can't attempt to acquire m while already holding conditionLock, because this would > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > >> already holding m, at line 265. > >> > >>> > >>> > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > >>> . > >>> . > >>> . > >>> 1 > >>> EnterCriticalSection(conditionLock); > >>> waitDone := (c.tickets # 0 AND c.counter # count); > >> A:---------^ > >> B:--------------------------------------------------^ > >>> LeaveCriticalSection(conditionLock); > >>> END; (* WHILE *) > >>> IF waitDone THEN > >>> alerted := FALSE; > >>> END; > >>> m.acquire(); > >>> 2 > >>> EnterCriticalSection(conditionLock); > >>> DEC(c.waiters); > >>> IF waitDone THEN esp. here. > >>> DEC(c.tickets); > >> > >> C:---------------^ > >>> lastWaiter := (c.tickets = 0); > >>> END; > >>> LeaveCriticalSection(conditionLock); > >>> > >>> > >>> That is, if we decrement tickets earlier > >>> within the first critical section, > >>> while we will still wake everyone, > >>> only one will decide waitDone and the rest will keep looping. > >>> A downside of this, perhaps, is that waking all for Broadcast > >>> might be a little slower. > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > >>> implementations, they seem to deal with this differently than us, > >>> and they each do about the same thing -- they use a counted semaphore. > >>> In the boost case, it appears they duplicate the semaphore for every > >>> notification generation, which seems expensive. > >>> > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > >>> Or they can do their own reference counting? > >>> jdk7 seems to looke like jdk6. > >>> The code is gone in jdk8 and I can't easily find the delete in history. > >>> The lock merging jdk does probably helps here too. > >>> In fact they merge #1 and #2 above as a result. > >>> But they still initially wake all threads for signal, not just broadcast. > >>> There is also the problem that all the event waits > >>> are followed by EnterCriticalSection (or jdk facsimile). > >>> - Jay > >>> > >>> > >>> > >>> > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! > >> --- > >>> From: jay.krell at cornell.edu > >>> To: m3devel at elegosoft.com > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > >>> > >>> Here is another implementation to consider: > >>> > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > >>> > >>> - Jay > >>> > >>> > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! > >> --- > >>> From: jay.krell at cornell.edu > >>> To: m3devel at elegosoft.com > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > >>> Subject: [M3devel] purported condition variable problems on Win32? > >>> > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > >>> > >>> vs. > >>> > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >>> vs. > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > >>> I wrote this: > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > >>> alertable: BOOLEAN) RAISES {Alerted} = > >>> (* LL = m on entry and exit, but not for the duration > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > >>> * NOTE that they merge the user lock and the condition lock. > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >>> * "3.3. The Generation Count Solution" > >>> *) > >>> > >>> Do we have the problems described in README.CV? > >>> > >>> I haven't looked through the ACE code to see > >>> to what extent they resemble solution 3.3, or if they > >>> changed as a result of this discussion -- which I admit I don't understand > >>> and haven't read closely. > >>> > >>> > >>> Spurious wakeups are ok, though should be minimized. > >>> > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > >>> > >>> Thank you, > >>> - Jay > >>> > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >>> > >>> > >>> _______________________________________________ > >>> M3devel mailing list > >>> M3devel at elegosoft.com > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >> > >> -- > >> Rodney Bates > >> rodney.m.bates at acm.org > >> > >> > > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Sat Jul 23 23:02:15 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sat, 23 Jul 2016 21:02:15 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop>, <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop>, , Message-ID: <5793DB37.10603@lcwb.coop> On 07/22/2016 03:20 AM, Jay K wrote: > Here is very weak circumstantial evidence that the locks can be merged: > > > > https://msdn.microsoft.com/en-us/library/windows/desktop/ms682052(v=vs.85).aspx > > "It is often convenient to use more than one condition variable with the same lock. > For example, an implementation of a reader/writer lock might use a single critical > section but separate condition variables for readers and writers. > " > > > i.e. the mapping isn't one lock to one condition, but the convenient > construct is multiple conditions to one lock, not multiple locks to one condition. > > > http://linux.die.net/man/3/pthread_cond_wait > > " > The effect of using more than one mutex for concurrent pthread_cond_timedwait() or > pthread_cond_wait() operations on the same condition variable is undefined; that is, > a condition variable becomes bound to a unique mutex when a thread waits on the condition > variable, and this (dynamic) binding shall end when the wait returns. > " > Yes, these are the usual rules about the relationship between mutexs and condition variables. There is sometimes a definite need for >1 condition variable associated with a single mutex. The other way around might be definable, but I think would create a big mess. Our Thread interface documents only that Wait(m,c) must have m locked at the time of call (and will return with it locked too, but not necessarily for the duration. Chapter 4 of SPwM3, "An Introduction to Programming with Threads" informally defines a _monitor_ as "some data, a mutex, and zero or more condition variables" and says further "A particular condition variable is always used in conjunction with the same mutex and its data". (SPwM3, 4.3.2, p93) It looks to me like the formal specification of Wait, Signal, and Broadcast in SRwM3, 5.3.3, p124 would allow a condition to simultaneously contain waiters that passed in different mutexes. Each thread remembers locally what mutex it wants to reacquire when it wakes up. I'm not sure we really want to commit to that. Client code's doing it would no doubt create even trickier code than this kind of stuff already is. Off hand, I have a hard time thinking of a reasonable use-case. If we decide to go with the informal statement, the implementations really should enforce it by storing the associated mutex in the condition variable, and verifying that additional waiters supply the same one. Probably, in most implementations, you could get away with moving the association of a condition variable from one mutex to another when no threads were waiting on the condition. Buy\t the only possible benefit I can think of would be memory savings, not needing as many condition variables, since the different mutex associations would be, well, mutually exclusive. > i.e. while the lock used with a condition variable can change through time, > there can be only one at a time. > > And then, presumably win32 condition variables and posix condition variables > and Modula-3 Thread.Condition are all kind of similar. > > Hm, ok, a problem with merging the locks, is it leaves no lock for Signal/Broadcast. > Perhaps they should use atomic/Interlocked. > > - Jay > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > From: jay.krell at cornell.edu > To: rodney.m.bates at acm.org > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] purported condition variable problems on Win32? > Date: Thu, 21 Jul 2016 06:43:16 +0000 > > That question was meant for the other bug. > > Can this one be fixed by rechecking if tickets is positive before decrementing it? > This seems like a big problem in Schmidt's code. > > I'm nervous about merging the locks, but maybe. > In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. > Though I realize that is actually necessarily part of what gets done. > Schmidt clearly did not merge the locks, and the Java version does. > The Java version also has its own simple and not-terrible critical section. > The Java version also disappeared in newer releases and I didn't track down what happened to it. > > - Jay > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > From: jay.krell at cornell.edu > To: rodney.m.bates at acm.org > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] purported condition variable problems on Win32? > Date: Thu, 21 Jul 2016 06:35:41 +0000 > > Am I understanding this? > There are some waiters. > There is a broadcast ("wake all"). Some waiters are woken. > There are more waiters. > There is a signal ("wake one"). > There is a chance that some of the original waiters will remain waiting while the later waiters are not. > That is, the signaled threads kind of steal the wake intended by the broadcast. > > > Is this unfairness or incorrectness? > It feels like unfairness. > It feels like condition variables are specified fairly weakly. > > > - Jay > > > > Date: Wed, 20 Jul 2016 11:43:27 -0500 > > From: rodney_bates at lcwb.coop > > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] purported condition variable problems on Win32? > > > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > > > This gets confusing, since there are Modula-3 waits, signals, > > etc. and similar concepts in Win32, but which are not the > > same. I am going the try to put "M3-" and "Win-" prefixes on > > things to make this clearer. I cartainly need to do this > > for my own benefit, if for nobody else. > > > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > > c.tickets = 0, and c.waiters = 2. > > > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > > which will allow both the waiters to Win-wakeup sometime soon. > > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > > to eventually allow only one waiter to M3-wakeup) and increments > > c.counter, setting things up so that the waitDone expression at :308 > > will be TRUE for the waiting threads, the next time they evaluate it. > > > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > > releases c.lock, and tries to acquire m. The problem arises > > because the second waiter can get to these steps before the first > > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > > second to reach :308 will find c.tickets # 0, as did the first, > > and proceed. > > > > Each will eventually, do a M3-wake. If this were Posix condition > > variables, this would be wrong, because Posix says only one > > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > > says one or more waiters proceed. However, each will, in the > > process, get to :322, and DEC(c.tickets). Signal only provided > > one ticket, but the waiters have taken two. c.tickets goes > > negative. > > > > After this, when some thread can do another (M3-)Wait, and some > > other does another (M3-)Signal, c.tickets increments only back to > > zero, which means the new waiter will not M3-wake, even though it > > should. > > > > > > I am going to try to construct a test case that will force > > this scenario. > > > > See more below: > > > > > > On 07/18/2016 08:53 PM, Jay wrote: > > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > > > Otherwise previous version I think had a very large lock. > > > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > > > I guess you are saying it would be unfair, which is ok. > > > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > > > I'll check on the unused event. > > > > > > - Jay > > > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > > >> > > >> > > >> > > >> > > >> > > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > > >> comments: > > >> > > >> This looks like a very direct implementation of Schmidts' generation count > > >> algorithm, with a number of important little details additionally taken care > > >> of. > > >> > > >> On small difference is at line 308 (B, below) where Schmidt does > > >> c.counter>count instead of #. Usually, this would make no difference, > > >> since counter only increases, but on a long-running program, it could > > >> overflow. If this happened,as it often does, as silent wrap-around to > > >> FIRST(INTEGER), some threads could be trapped for an extremely long > > >> time, waiting for counter come back around greater than their count. As > > >> coded in ThreadWin32, they would be allowed to proceed normally. > > >> > > >> The only glitch would be that, if there were simultaneously waiting threads > > >> separated by NUMBER(INTEGER) truly different generations (but the same value > > >> of c.counter), the recent ones could unfairly proceed along with the > > >> extremely old ones. That seems extremely less likely than the > > >> mere existence of an overflow. > > >> > > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > > >> only created at :537 and deleted at :554, but never used in any real > > >> way. It could be deleted. > > >> > > >> As for this code: > > >> > > >> EnterCriticalSection(conditionLock); > > >> > > >> (* Capture the value of the counter before we start waiting. > > >> * We will not stop waiting until the counter changes. > > >> * That is, we will not stop waiting until a signal > > >> * comes in after we start waiting. > > >> *) > > >> > > >> count := c.counter; > > >> INC(c.waiters); > > >> > > >> LeaveCriticalSection(conditionLock); > > >> m.release(); (* can this be moved to before taking conditionLock? *) > > >> > > >> No, it is important to sample and save c.counter in the order threads wait. > > >> Retaining m until this is done ensures this. Otherwise, some other thread > > >> that waited later could have altered c.counter before this one gets conditionLock > > >> and saves its copy of c.counter. > > >> > > >> I have made several comment changes to ThreadWin32.m3 that would have made > > >> it easier to vet. I will commit these, but only comments. It is either > > >> very difficult or impossible for me to test or even recompile this module, > > >> and, especially as fragile as this kind of code is, I don't want to commit > > >> any substantive changes untested and uncompiled. > > >> > > >> More below: > > >> > > >>> On 07/07/2016 04:24 AM, Jay K wrote: > > >>> So...I do NOT understand all of this. > > >>> > > >>> > > >>> However, comparing the three implementations > > >>> and attempting to understand ours, > > >>> I am struck by the following > > >>> > > >>> - Our broadcast seems ok, but > > >>> - our signal seems to wake everyone, > > >>> and then...they race a bit, > > >>> one will decide it is last, and > > >>> reset the event, but every prior waiter > > >>> is still woken. > > >>> Why not merge the following chunks: > > >> > > >> I don't think this will work. It is important that, once a thread has decided, > > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > > >> if it is also the last of its generation to proceed, resets c.waitevent. This > > >> is ensured by its having already reacquired m first, preventing any other thread > > >> from returning from its Wait yet. > > > > I was wrong about this. There is already no fairness being enforced here. Even with > > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > > > Merging, as you proposed, is probably close to the fix for this bug. > > > > > > >> > > >> And it can't attempt to acquire m while already holding conditionLock, because this would > > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > > >> already holding m, at line 265. > > >> > > >>> > > >>> > > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > > >>> . > > >>> . > > >>> . > > >>> 1 > > >>> EnterCriticalSection(conditionLock); > > >>> waitDone := (c.tickets # 0 AND c.counter # count); > > >> A:---------^ > > >> B:--------------------------------------------------^ > > >>> LeaveCriticalSection(conditionLock); > > >>> END; (* WHILE *) > > >>> IF waitDone THEN > > >>> alerted := FALSE; > > >>> END; > > >>> m.acquire(); > > >>> 2 > > >>> EnterCriticalSection(conditionLock); > > >>> DEC(c.waiters); > > >>> IF waitDone THEN esp. here. > > >>> DEC(c.tickets); > > >> > > >> C:---------------^ > > >>> lastWaiter := (c.tickets = 0); > > >>> END; > > >>> LeaveCriticalSection(conditionLock); > > >>> > > >>> > > >>> That is, if we decrement tickets earlier > > >>> within the first critical section, > > >>> while we will still wake everyone, > > >>> only one will decide waitDone and the rest will keep looping. > > >>> A downside of this, perhaps, is that waking all for Broadcast > > >>> might be a little slower. > > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > > >>> implementations, they seem to deal with this differently than us, > > >>> and they each do about the same thing -- they use a counted semaphore. > > >>> In the boost case, it appears they duplicate the semaphore for every > > >>> notification generation, which seems expensive. > > >>> > > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > > >>> Or they can do their own reference counting? > > >>> jdk7 seems to looke like jdk6. > > >>> The code is gone in jdk8 and I can't easily find the delete in history. > > >>> The lock merging jdk does probably helps here too. > > >>> In fact they merge #1 and #2 above as a result. > > >>> But they still initially wake all threads for signal, not just broadcast. > > >>> There is also the problem that all the event waits > > >>> are followed by EnterCriticalSection (or jdk facsimile). > > >>> - Jay > > >>> > > >>> > > >>> > > >>> > > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > --! > > >> --- > > >>> From: jay.krell at cornell.edu > > >>> To: m3devel at elegosoft.com > > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > > >>> > > >>> Here is another implementation to consider: > > >>> > > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > > >>> > > >>> - Jay > > >>> > > >>> > > >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > --! > > >> --- > > >>> From: jay.krell at cornell.edu > > >>> To: m3devel at elegosoft.com > > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > > >>> Subject: [M3devel] purported condition variable problems on Win32? > > >>> > > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > > >>> > > >>> vs. > > >>> > > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > >>> vs. > > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > > >>> I wrote this: > > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > > >>> alertable: BOOLEAN) RAISES {Alerted} = > > >>> (* LL = m on entry and exit, but not for the duration > > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > > >>> * NOTE that they merge the user lock and the condition lock. > > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > >>> * "3.3. The Generation Count Solution" > > >>> *) > > >>> > > >>> Do we have the problems described in README.CV? > > >>> > > >>> I haven't looked through the ACE code to see > > >>> to what extent they resemble solution 3.3, or if they > > >>> changed as a result of this discussion -- which I admit I don't understand > > >>> and haven't read closely. > > >>> > > >>> > > >>> Spurious wakeups are ok, though should be minimized. > > >>> > > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > >>> > > >>> Thank you, > > >>> - Jay > > >>> > > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > >>> > > >>> > > >>> _______________________________________________ > > >>> M3devel mailing list > > >>> M3devel at elegosoft.com > > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > >> > > >> -- > > >> Rodney Bates > > >> rodney.m.bates at acm.org > > >> > > >> > > > > > > > -- > > Rodney Bates > > rodney.m.bates at acm.org -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Sun Jul 24 01:37:14 2016 From: jay.krell at cornell.edu (Jay K) Date: Sat, 23 Jul 2016 23:37:14 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: <5793DB37.10603@lcwb.coop> References: <578BF33B.2030605@lcwb.coop>, <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop>, , , , <5793DB37.10603@lcwb.coop> Message-ID: I don't think Signal/Broadcast require caller be holding any lock, is a problem. - Jay > Date: Sat, 23 Jul 2016 16:01:43 -0500 > From: rodney_bates at lcwb.coop > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] purported condition variable problems on Win32? > > > > On 07/22/2016 03:20 AM, Jay K wrote: > > Here is very weak circumstantial evidence that the locks can be merged: > > > > > > > > https://msdn.microsoft.com/en-us/library/windows/desktop/ms682052(v=vs.85).aspx > > > > "It is often convenient to use more than one condition variable with the same lock. > > For example, an implementation of a reader/writer lock might use a single critical > > section but separate condition variables for readers and writers. > > " > > > > > > i.e. the mapping isn't one lock to one condition, but the convenient > > construct is multiple conditions to one lock, not multiple locks to one condition. > > > > > > http://linux.die.net/man/3/pthread_cond_wait > > > > " > > The effect of using more than one mutex for concurrent pthread_cond_timedwait() or > > pthread_cond_wait() operations on the same condition variable is undefined; that is, > > a condition variable becomes bound to a unique mutex when a thread waits on the condition > > variable, and this (dynamic) binding shall end when the wait returns. > > " > > > > Yes, these are the usual rules about the relationship between mutexs and condition variables. > There is sometimes a definite need for >1 condition variable associated with a single > mutex. The other way around might be definable, but I think would create a big mess. > > Our Thread interface documents only that Wait(m,c) must have m locked at the time of > call (and will return with it locked too, but not necessarily for the duration. > > Chapter 4 of SPwM3, "An Introduction to Programming with Threads" informally defines > a _monitor_ as "some data, a mutex, and zero or more condition variables" and says > further "A particular condition variable is always used in conjunction with the same > mutex and its data". (SPwM3, 4.3.2, p93) > > It looks to me like the formal specification of Wait, Signal, and Broadcast in > SRwM3, 5.3.3, p124 would allow a condition to simultaneously contain waiters > that passed in different mutexes. Each thread remembers locally what mutex it > wants to reacquire when it wakes up. > > I'm not sure we really want to commit to that. Client code's doing it would no doubt > create even trickier code than this kind of stuff already is. Off hand, I have a > hard time thinking of a reasonable use-case. > > If we decide to go with the informal statement, the implementations really should > enforce it by storing the associated mutex in the condition variable, and verifying > that additional waiters supply the same one. > > Probably, in most implementations, you could get away with moving the association > of a condition variable from one mutex to another when no threads were waiting on > the condition. Buy\t the only possible benefit I can think of would be memory savings, > not needing as many condition variables, since the different mutex associations would > be, well, mutually exclusive. > > > > > > i.e. while the lock used with a condition variable can change through time, > > there can be only one at a time. > > > > And then, presumably win32 condition variables and posix condition variables > > and Modula-3 Thread.Condition are all kind of similar. > > > > Hm, ok, a problem with merging the locks, is it leaves no lock for Signal/Broadcast. > > Perhaps they should use atomic/Interlocked. > > > > - Jay > > > > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --- > > From: jay.krell at cornell.edu > > To: rodney.m.bates at acm.org > > CC: m3devel at elegosoft.com > > Subject: RE: [M3devel] purported condition variable problems on Win32? > > Date: Thu, 21 Jul 2016 06:43:16 +0000 > > > > That question was meant for the other bug. > > > > Can this one be fixed by rechecking if tickets is positive before decrementing it? > > This seems like a big problem in Schmidt's code. > > > > I'm nervous about merging the locks, but maybe. > > In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. > > Though I realize that is actually necessarily part of what gets done. > > Schmidt clearly did not merge the locks, and the Java version does. > > The Java version also has its own simple and not-terrible critical section. > > The Java version also disappeared in newer releases and I didn't track down what happened to it. > > > > - Jay > > > > > > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --- > > From: jay.krell at cornell.edu > > To: rodney.m.bates at acm.org > > CC: m3devel at elegosoft.com > > Subject: RE: [M3devel] purported condition variable problems on Win32? > > Date: Thu, 21 Jul 2016 06:35:41 +0000 > > > > Am I understanding this? > > There are some waiters. > > There is a broadcast ("wake all"). Some waiters are woken. > > There are more waiters. > > There is a signal ("wake one"). > > There is a chance that some of the original waiters will remain waiting while the later waiters are not. > > That is, the signaled threads kind of steal the wake intended by the broadcast. > > > > > > Is this unfairness or incorrectness? > > It feels like unfairness. > > It feels like condition variables are specified fairly weakly. > > > > > > - Jay > > > > > > > Date: Wed, 20 Jul 2016 11:43:27 -0500 > > > From: rodney_bates at lcwb.coop > > > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > > > CC: m3devel at elegosoft.com > > > Subject: Re: [M3devel] purported condition variable problems on Win32? > > > > > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > > > > > This gets confusing, since there are Modula-3 waits, signals, > > > etc. and similar concepts in Win32, but which are not the > > > same. I am going the try to put "M3-" and "Win-" prefixes on > > > things to make this clearer. I cartainly need to do this > > > for my own benefit, if for nobody else. > > > > > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > > > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > > > c.tickets = 0, and c.waiters = 2. > > > > > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > > > which will allow both the waiters to Win-wakeup sometime soon. > > > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > > > to eventually allow only one waiter to M3-wakeup) and increments > > > c.counter, setting things up so that the waitDone expression at :308 > > > will be TRUE for the waiting threads, the next time they evaluate it. > > > > > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > > > releases c.lock, and tries to acquire m. The problem arises > > > because the second waiter can get to these steps before the first > > > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > > > second to reach :308 will find c.tickets # 0, as did the first, > > > and proceed. > > > > > > Each will eventually, do a M3-wake. If this were Posix condition > > > variables, this would be wrong, because Posix says only one > > > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > > > says one or more waiters proceed. However, each will, in the > > > process, get to :322, and DEC(c.tickets). Signal only provided > > > one ticket, but the waiters have taken two. c.tickets goes > > > negative. > > > > > > After this, when some thread can do another (M3-)Wait, and some > > > other does another (M3-)Signal, c.tickets increments only back to > > > zero, which means the new waiter will not M3-wake, even though it > > > should. > > > > > > > > > I am going to try to construct a test case that will force > > > this scenario. > > > > > > See more below: > > > > > > > > > On 07/18/2016 08:53 PM, Jay wrote: > > > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > > > > > Otherwise previous version I think had a very large lock. > > > > > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > > > > > I guess you are saying it would be unfair, which is ok. > > > > > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > > > > > I'll check on the unused event. > > > > > > > > - Jay > > > > > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > > > >> comments: > > > >> > > > >> This looks like a very direct implementation of Schmidts' generation count > > > >> algorithm, with a number of important little details additionally taken care > > > >> of. > > > >> > > > >> On small difference is at line 308 (B, below) where Schmidt does > > > >> c.counter>count instead of #. Usually, this would make no difference, > > > >> since counter only increases, but on a long-running program, it could > > > >> overflow. If this happened,as it often does, as silent wrap-around to > > > >> FIRST(INTEGER), some threads could be trapped for an extremely long > > > >> time, waiting for counter come back around greater than their count. As > > > >> coded in ThreadWin32, they would be allowed to proceed normally. > > > >> > > > >> The only glitch would be that, if there were simultaneously waiting threads > > > >> separated by NUMBER(INTEGER) truly different generations (but the same value > > > >> of c.counter), the recent ones could unfairly proceed along with the > > > >> extremely old ones. That seems extremely less likely than the > > > >> mere existence of an overflow. > > > >> > > > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > > > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > > > >> only created at :537 and deleted at :554, but never used in any real > > > >> way. It could be deleted. > > > >> > > > >> As for this code: > > > >> > > > >> EnterCriticalSection(conditionLock); > > > >> > > > >> (* Capture the value of the counter before we start waiting. > > > >> * We will not stop waiting until the counter changes. > > > >> * That is, we will not stop waiting until a signal > > > >> * comes in after we start waiting. > > > >> *) > > > >> > > > >> count := c.counter; > > > >> INC(c.waiters); > > > >> > > > >> LeaveCriticalSection(conditionLock); > > > >> m.release(); (* can this be moved to before taking conditionLock? *) > > > >> > > > >> No, it is important to sample and save c.counter in the order threads wait. > > > >> Retaining m until this is done ensures this. Otherwise, some other thread > > > >> that waited later could have altered c.counter before this one gets conditionLock > > > >> and saves its copy of c.counter. > > > >> > > > >> I have made several comment changes to ThreadWin32.m3 that would have made > > > >> it easier to vet. I will commit these, but only comments. It is either > > > >> very difficult or impossible for me to test or even recompile this module, > > > >> and, especially as fragile as this kind of code is, I don't want to commit > > > >> any substantive changes untested and uncompiled. > > > >> > > > >> More below: > > > >> > > > >>> On 07/07/2016 04:24 AM, Jay K wrote: > > > >>> So...I do NOT understand all of this. > > > >>> > > > >>> > > > >>> However, comparing the three implementations > > > >>> and attempting to understand ours, > > > >>> I am struck by the following > > > >>> > > > >>> - Our broadcast seems ok, but > > > >>> - our signal seems to wake everyone, > > > >>> and then...they race a bit, > > > >>> one will decide it is last, and > > > >>> reset the event, but every prior waiter > > > >>> is still woken. > > > >>> Why not merge the following chunks: > > > >> > > > >> I don't think this will work. It is important that, once a thread has decided, > > > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > > > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > > > >> if it is also the last of its generation to proceed, resets c.waitevent. This > > > >> is ensured by its having already reacquired m first, preventing any other thread > > > >> from returning from its Wait yet. > > > > > > I was wrong about this. There is already no fairness being enforced here. Even with > > > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > > > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > > > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > > > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > > > > > Merging, as you proposed, is probably close to the fix for this bug. > > > > > > > > > >> > > > >> And it can't attempt to acquire m while already holding conditionLock, because this would > > > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > > > >> already holding m, at line 265. > > > >> > > > >>> > > > >>> > > > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > > > >>> . > > > >>> . > > > >>> . > > > >>> 1 > > > >>> EnterCriticalSection(conditionLock); > > > >>> waitDone := (c.tickets # 0 AND c.counter # count); > > > >> A:---------^ > > > >> B:--------------------------------------------------^ > > > >>> LeaveCriticalSection(conditionLock); > > > >>> END; (* WHILE *) > > > >>> IF waitDone THEN > > > >>> alerted := FALSE; > > > >>> END; > > > >>> m.acquire(); > > > >>> 2 > > > >>> EnterCriticalSection(conditionLock); > > > >>> DEC(c.waiters); > > > >>> IF waitDone THEN esp. here. > > > >>> DEC(c.tickets); > > > >> > > > >> C:---------------^ > > > >>> lastWaiter := (c.tickets = 0); > > > >>> END; > > > >>> LeaveCriticalSection(conditionLock); > > > >>> > > > >>> > > > >>> That is, if we decrement tickets earlier > > > >>> within the first critical section, > > > >>> while we will still wake everyone, > > > >>> only one will decide waitDone and the rest will keep looping. > > > >>> A downside of this, perhaps, is that waking all for Broadcast > > > >>> might be a little slower. > > > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > > > >>> implementations, they seem to deal with this differently than us, > > > >>> and they each do about the same thing -- they use a counted semaphore. > > > >>> In the boost case, it appears they duplicate the semaphore for every > > > >>> notification generation, which seems expensive. > > > >>> > > > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > > > >>> Or they can do their own reference counting? > > > >>> jdk7 seems to looke like jdk6. > > > >>> The code is gone in jdk8 and I can't easily find the delete in history. > > > >>> The lock merging jdk does probably helps here too. > > > >>> In fact they merge #1 and #2 above as a result. > > > >>> But they still initially wake all threads for signal, not just broadcast. > > > >>> There is also the problem that all the event waits > > > >>> are followed by EnterCriticalSection (or jdk facsimile). > > > >>> - Jay > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > -----! > > > --! > > > >> --- > > > >>> From: jay.krell at cornell.edu > > > >>> To: m3devel at elegosoft.com > > > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > > > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > > > >>> > > > >>> Here is another implementation to consider: > > > >>> > > > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > > > >>> > > > >>> - Jay > > > >>> > > > >>> > > > >>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > -----! > > > --! > > > >> --- > > > >>> From: jay.krell at cornell.edu > > > >>> To: m3devel at elegosoft.com > > > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > > > >>> Subject: [M3devel] purported condition variable problems on Win32? > > > >>> > > > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > > > >>> > > > >>> vs. > > > >>> > > > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > > >>> vs. > > > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > > > >>> I wrote this: > > > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > > > >>> alertable: BOOLEAN) RAISES {Alerted} = > > > >>> (* LL = m on entry and exit, but not for the duration > > > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > > > >>> * NOTE that they merge the user lock and the condition lock. > > > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > > >>> * "3.3. The Generation Count Solution" > > > >>> *) > > > >>> > > > >>> Do we have the problems described in README.CV? > > > >>> > > > >>> I haven't looked through the ACE code to see > > > >>> to what extent they resemble solution 3.3, or if they > > > >>> changed as a result of this discussion -- which I admit I don't understand > > > >>> and haven't read closely. > > > >>> > > > >>> > > > >>> Spurious wakeups are ok, though should be minimized. > > > >>> > > > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > > >>> > > > >>> Thank you, > > > >>> - Jay > > > >>> > > > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > > >>> > > > >>> > > > >>> _______________________________________________ > > > >>> M3devel mailing list > > > >>> M3devel at elegosoft.com > > > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > > >> > > > >> -- > > > >> Rodney Bates > > > >> rodney.m.bates at acm.org > > > >> > > > >> > > > > > > > > > > -- > > > Rodney Bates > > > rodney.m.bates at acm.org > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Sun Jul 24 18:29:34 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 24 Jul 2016 16:29:34 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop>, <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com>, <578FAA2F.6020105@lcwb.coop>, , , , <5793DB37.10603@lcwb.coop> Message-ID: <5794ECAE.5070308@lcwb.coop> No, they don't. But I don't think that presents a problem. On 07/23/2016 06:37 PM, Jay K wrote: > I don't think Signal/Broadcast require caller be holding any lock, is a problem. > > - Jay > > > > > Date: Sat, 23 Jul 2016 16:01:43 -0500 > > From: rodney_bates at lcwb.coop > > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] purported condition variable problems on Win32? > > > > > > > > On 07/22/2016 03:20 AM, Jay K wrote: > > > Here is very weak circumstantial evidence that the locks can be merged: > > > > > > > > > > > > https://msdn.microsoft.com/en-us/library/windows/desktop/ms682052(v=vs.85).aspx > > > > > > "It is often convenient to use more than one condition variable with the same lock. > > > For example, an implementation of a reader/writer lock might use a single critical > > > section but separate condition variables for readers and writers. > > > " > > > > > > > > > i.e. the mapping isn't one lock to one condition, but the convenient > > > construct is multiple conditions to one lock, not multiple locks to one condition. > > > > > > > > > http://linux.die.net/man/3/pthread_cond_wait > > > > > > " > > > The effect of using more than one mutex for concurrent pthread_cond_timedwait() or > > > pthread_cond_wait() operations on the same condition variable is undefined; that is, > > > a condition variable becomes bound to a unique mutex when a thread waits on the condition > > > variable, and this (dynamic) binding shall end when the wait returns. > > > " > > > > > > > Yes, these are the usual rules about the relationship between mutexs and condition variables. > > There is sometimes a definite need for >1 condition variable associated with a single > > mutex. The other way around might be definable, but I think would create a big mess. > > > > Our Thread interface documents only that Wait(m,c) must have m locked at the time of > > call (and will return with it locked too, but not necessarily for the duration. > > > > Chapter 4 of SPwM3, "An Introduction to Programming with Threads" informally defines > > a _monitor_ as "some data, a mutex, and zero or more condition variables" and says > > further "A particular condition variable is always used in conjunction with the same > > mutex and its data". (SPwM3, 4.3.2, p93) > > > > It looks to me like the formal specification of Wait, Signal, and Broadcast in > > SRwM3, 5.3.3, p124 would allow a condition to simultaneously contain waiters > > that passed in different mutexes. Each thread remembers locally what mutex it > > wants to reacquire when it wakes up. > > > > I'm not sure we really want to commit to that. Client code's doing it would no doubt > > create even trickier code than this kind of stuff already is. Off hand, I have a > > hard time thinking of a reasonable use-case. > > > > If we decide to go with the informal statement, the implementations really should > > enforce it by storing the associated mutex in the condition variable, and verifying > > that additional waiters supply the same one. > > > > Probably, in most implementations, you could get away with moving the association > > of a condition variable from one mutex to another when no threads were waiting on > > the condition. Buy\t the only possible benefit I can think of would be memory savings, > > not needing as many condition variables, since the different mutex associations would > > be, well, mutually exclusive. > > > > > > > > > > > i.e. while the lock used with a condition variable can change through time, > > > there can be only one at a time. > > > > > > And then, presumably win32 condition variables and posix condition variables > > > and Modula-3 Thread.Condition are all kind of similar. > > > > > > Hm, ok, a problem with merging the locks, is it leaves no lock for Signal/Broadcast. > > > Perhaps they should use atomic/Interlocked. > > > > > > - Jay > > > > > > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > --- > > > From: jay.krell at cornell.edu > > > To: rodney.m.bates at acm.org > > > CC: m3devel at elegosoft.com > > > Subject: RE: [M3devel] purported condition variable problems on Win32? > > > Date: Thu, 21 Jul 2016 06:43:16 +0000 > > > > > > That question was meant for the other bug. > > > > > > Can this one be fixed by rechecking if tickets is positive before decrementing it? > > > This seems like a big problem in Schmidt's code. > > > > > > I'm nervous about merging the locks, but maybe. > > > In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. > > > Though I realize that is actually necessarily part of what gets done. > > > Schmidt clearly did not merge the locks, and the Java version does. > > > The Java version also has its own simple and not-terrible critical section. > > > The Java version also disappeared in newer releases and I didn't track down what happened to it. > > > > > > - Jay > > > > > > > > > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > --- > > > From: jay.krell at cornell.edu > > > To: rodney.m.bates at acm.org > > > CC: m3devel at elegosoft.com > > > Subject: RE: [M3devel] purported condition variable problems on Win32? > > > Date: Thu, 21 Jul 2016 06:35:41 +0000 > > > > > > Am I understanding this? > > > There are some waiters. > > > There is a broadcast ("wake all"). Some waiters are woken. > > > There are more waiters. > > > There is a signal ("wake one"). > > > There is a chance that some of the original waiters will remain waiting while the later waiters are not. > > > That is, the signaled threads kind of steal the wake intended by the broadcast. > > > > > > > > > Is this unfairness or incorrectness? > > > It feels like unfairness. > > > It feels like condition variables are specified fairly weakly. > > > > > > > > > - Jay > > > > > > > > > > Date: Wed, 20 Jul 2016 11:43:27 -0500 > > > > From: rodney_bates at lcwb.coop > > > > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > > > > CC: m3devel at elegosoft.com > > > > Subject: Re: [M3devel] purported condition variable problems on Win32? > > > > > > > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > > > > > > > > This gets confusing, since there are Modula-3 waits, signals, > > > > etc. and similar concepts in Win32, but which are not the > > > > same. I am going the try to put "M3-" and "Win-" prefixes on > > > > things to make this clearer. I cartainly need to do this > > > > for my own benefit, if for nobody else. > > > > > > > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > > > > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > > > > c.tickets = 0, and c.waiters = 2. > > > > > > > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > > > > which will allow both the waiters to Win-wakeup sometime soon. > > > > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > > > > to eventually allow only one waiter to M3-wakeup) and increments > > > > c.counter, setting things up so that the waitDone expression at :308 > > > > will be TRUE for the waiting threads, the next time they evaluate it. > > > > > > > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > > > > releases c.lock, and tries to acquire m. The problem arises > > > > because the second waiter can get to these steps before the first > > > > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > > > > second to reach :308 will find c.tickets # 0, as did the first, > > > > and proceed. > > > > > > > > Each will eventually, do a M3-wake. If this were Posix condition > > > > variables, this would be wrong, because Posix says only one > > > > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > > > > says one or more waiters proceed. However, each will, in the > > > > process, get to :322, and DEC(c.tickets). Signal only provided > > > > one ticket, but the waiters have taken two. c.tickets goes > > > > negative. > > > > > > > > After this, when some thread can do another (M3-)Wait, and some > > > > other does another (M3-)Signal, c.tickets increments only back to > > > > zero, which means the new waiter will not M3-wake, even though it > > > > should. > > > > > > > > > > > > I am going to try to construct a test case that will force > > > > this scenario. > > > > > > > > See more below: > > > > > > > > > > > > On 07/18/2016 08:53 PM, Jay wrote: > > > > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > > > > > > > > > > Otherwise previous version I think had a very large lock. > > > > > > > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > > > > > > > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > > > > > > > > > > I guess you are saying it would be unfair, which is ok. > > > > > > > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > > > > > > > > > > I'll check on the unused event. > > > > > > > > > > - Jay > > > > > > > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > > > > >> comments: > > > > >> > > > > >> This looks like a very direct implementation of Schmidts' generation count > > > > >> algorithm, with a number of important little details additionally taken care > > > > >> of. > > > > >> > > > > >> On small difference is at line 308 (B, below) where Schmidt does > > > > >> c.counter>count instead of #. Usually, this would make no difference, > > > > >> since counter only increases, but on a long-running program, it could > > > > >> overflow. If this happened,as it often does, as silent wrap-around to > > > > >> FIRST(INTEGER), some threads could be trapped for an extremely long > > > > >> time, waiting for counter come back around greater than their count. As > > > > >> coded in ThreadWin32, they would be allowed to proceed normally. > > > > >> > > > > >> The only glitch would be that, if there were simultaneously waiting threads > > > > >> separated by NUMBER(INTEGER) truly different generations (but the same value > > > > >> of c.counter), the recent ones could unfairly proceed along with the > > > > >> extremely old ones. That seems extremely less likely than the > > > > >> mere existence of an overflow. > > > > >> > > > > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > > > > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > > > > >> only created at :537 and deleted at :554, but never used in any real > > > > >> way. It could be deleted. > > > > >> > > > > >> As for this code: > > > > >> > > > > >> EnterCriticalSection(conditionLock); > > > > >> > > > > >> (* Capture the value of the counter before we start waiting. > > > > >> * We will not stop waiting until the counter changes. > > > > >> * That is, we will not stop waiting until a signal > > > > >> * comes in after we start waiting. > > > > >> *) > > > > >> > > > > >> count := c.counter; > > > > >> INC(c.waiters); > > > > >> > > > > >> LeaveCriticalSection(conditionLock); > > > > >> m.release(); (* can this be moved to before taking conditionLock? *) > > > > >> > > > > >> No, it is important to sample and save c.counter in the order threads wait. > > > > >> Retaining m until this is done ensures this. Otherwise, some other thread > > > > >> that waited later could have altered c.counter before this one gets conditionLock > > > > >> and saves its copy of c.counter. > > > > >> > > > > >> I have made several comment changes to ThreadWin32.m3 that would have made > > > > >> it easier to vet. I will commit these, but only comments. It is either > > > > >> very difficult or impossible for me to test or even recompile this module, > > > > >> and, especially as fragile as this kind of code is, I don't want to commit > > > > >> any substantive changes untested and uncompiled. > > > > >> > > > > >> More below: > > > > >> > > > > >>> On 07/07/2016 04:24 AM, Jay K wrote: > > > > >>> So...I do NOT understand all of this. > > > > >>> > > > > >>> > > > > >>> However, comparing the three implementations > > > > >>> and attempting to understand ours, > > > > >>> I am struck by the following > > > > >>> > > > > >>> - Our broadcast seems ok, but > > > > >>> - our signal seems to wake everyone, > > > > >>> and then...they race a bit, > > > > >>> one will decide it is last, and > > > > >>> reset the event, but every prior waiter > > > > >>> is still woken. > > > > >>> Why not merge the following chunks: > > > > >> > > > > >> I don't think this will work. It is important that, once a thread has decided, > > > > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > > > > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > > > > >> if it is also the last of its generation to proceed, resets c.waitevent. This > > > > >> is ensured by its having already reacquired m first, preventing any other thread > > > > >> from returning from its Wait yet. > > > > > > > > I was wrong about this. There is already no fairness being enforced here. Even with > > > > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > > > > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > > > > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > > > > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > > > > > > > > Merging, as you proposed, is probably close to the fix for this bug. > > > > > > > > > > > > >> > > > > >> And it can't attempt to acquire m while already holding conditionLock, because this would > > > > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > > > > >> already holding m, at line 265. > > > > >> > > > > >>> > > > > >>> > > > > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > > > > >>> . > > > > >>> . > > > > >>> . > > > > >>> 1 > > > > >>> EnterCriticalSection(conditionLock); > > > > >>> waitDone := (c.tickets # 0 AND c.counter # count); > > > > >> A:---------^ > > > > >> B:--------------------------------------------------^ > > > > >>> LeaveCriticalSection(conditionLock); > > > > >>> END; (* WHILE *) > > > > >>> IF waitDone THEN > > > > >>> alerted := FALSE; > > > > >>> END; > > > > >>> m.acquire(); > > > > >>> 2 > > > > >>> EnterCriticalSection(conditionLock); > > > > >>> DEC(c.waiters); > > > > >>> IF waitDone THEN esp. here. > > > > >>> DEC(c.tickets); > > > > >> > > > > >> C:---------------^ > > > > >>> lastWaiter := (c.tickets = 0); > > > > >>> END; > > > > >>> LeaveCriticalSection(conditionLock); > > > > >>> > > > > >>> > > > > >>> That is, if we decrement tickets earlier > > > > >>> within the first critical section, > > > > >>> while we will still wake everyone, > > > > >>> only one will decide waitDone and the rest will keep looping. > > > > >>> A downside of this, perhaps, is that waking all for Broadcast > > > > >>> might be a little slower. > > > > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > > > > >>> implementations, they seem to deal with this differently than us, > > > > >>> and they each do about the same thing -- they use a counted semaphore. > > > > >>> In the boost case, it appears they duplicate the semaphore for every > > > > >>> notification generation, which seems expensive. > > > > >>> > > > > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > > > > >>> Or they can do their own reference counting? > > > > >>> jdk7 seems to looke like jdk6. > > > > >>> The code is gone in jdk8 and I can't easily find the delete in history. > > > > >>> The lock merging jdk does probably helps here too. > > > > >>> In fact they merge #1 and #2 above as a result. > > > > >>> But they still initially wake all threads for signal, not just broadcast. > > > > >>> There is also the problem that all the event waits > > > > >>> are followed by EnterCriticalSection (or jdk facsimile). > > > > >>> - Jay > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > -----! > > > > --! > > > > >> --- > > > > >>> From: jay.krell at cornell.edu > > > > >>> To: m3devel at elegosoft.com > > > > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > > > > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > > > > >>> > > > > >>> Here is another implementation to consider: > > > > >>> > > > > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > > > > >>> > > > > >>> - Jay > > > > >>> > > > > >>> > > > > >>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > -----! > > > > --! > > > > >> --- > > > > >>> From: jay.krell at cornell.edu > > > > >>> To: m3devel at elegosoft.com > > > > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > > > > >>> Subject: [M3devel] purported condition variable problems on Win32? > > > > >>> > > > > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > > > > >>> > > > > >>> vs. > > > > >>> > > > > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > > > >>> vs. > > > > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > > > > >>> I wrote this: > > > > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > > > > >>> alertable: BOOLEAN) RAISES {Alerted} = > > > > >>> (* LL = m on entry and exit, but not for the duration > > > > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > > > > >>> * NOTE that they merge the user lock and the condition lock. > > > > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > > > > >>> * "3.3. The Generation Count Solution" > > > > >>> *) > > > > >>> > > > > >>> Do we have the problems described in README.CV? > > > > >>> > > > > >>> I haven't looked through the ACE code to see > > > > >>> to what extent they resemble solution 3.3, or if they > > > > >>> changed as a result of this discussion -- which I admit I don't understand > > > > >>> and haven't read closely. > > > > >>> > > > > >>> > > > > >>> Spurious wakeups are ok, though should be minimized. > > > > >>> > > > > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > > > > >>> > > > > >>> Thank you, > > > > >>> - Jay > > > > >>> > > > > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > > > >>> > > > > >>> > > > > >>> _______________________________________________ > > > > >>> M3devel mailing list > > > > >>> M3devel at elegosoft.com > > > > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > > > >> > > > > >> -- > > > > >> Rodney Bates > > > > >> rodney.m.bates at acm.org > > > > >> > > > > >> > > > > > > > > > > > > > -- > > > > Rodney Bates > > > > rodney.m.bates at acm.org > > > > -- > > Rodney Bates > > rodney.m.bates at acm.org -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Sun Jul 24 23:53:20 2016 From: jay.krell at cornell.edu (Jay) Date: Sun, 24 Jul 2016 21:53:20 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: <5794ECAE.5070308@lcwb.coop> References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop> <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> <578FAA2F.6020105@lcwb.coop> <5793DB37.10603@lcwb.coop> <5794ECAE.5070308@lcwb.coop> Message-ID: It is a problem for merging mutex with condition lock, or rather, for having Wait rely on the external lock. - Jay > On Jul 24, 2016, at 9:28 AM, "Rodney M. Bates" wrote: > > No, they don't. But I don't think that presents a problem. > >> On 07/23/2016 06:37 PM, Jay K wrote: >> I don't think Signal/Broadcast require caller be holding any lock, is a problem. >> >> - Jay >> >> >> >> > Date: Sat, 23 Jul 2016 16:01:43 -0500 >> > From: rodney_bates at lcwb.coop >> > To: jay.krell at cornell.edu; rodney.m.bates at acm.org >> > CC: m3devel at elegosoft.com >> > Subject: Re: [M3devel] purported condition variable problems on Win32? >> > >> > >> > >> > On 07/22/2016 03:20 AM, Jay K wrote: >> > > Here is very weak circumstantial evidence that the locks can be merged: >> > > >> > > >> > > >> > > https://msdn.microsoft.com/en-us/library/windows/desktop/ms682052(v=vs.85).aspx >> > > >> > > "It is often convenient to use more than one condition variable with the same lock. >> > > For example, an implementation of a reader/writer lock might use a single critical >> > > section but separate condition variables for readers and writers. >> > > " >> > > >> > > >> > > i.e. the mapping isn't one lock to one condition, but the convenient >> > > construct is multiple conditions to one lock, not multiple locks to one condition. >> > > >> > > >> > > http://linux.die.net/man/3/pthread_cond_wait >> > > >> > > " >> > > The effect of using more than one mutex for concurrent pthread_cond_timedwait() or >> > > pthread_cond_wait() operations on the same condition variable is undefined; that is, >> > > a condition variable becomes bound to a unique mutex when a thread waits on the condition >> > > variable, and this (dynamic) binding shall end when the wait returns. >> > > " >> > > >> > >> > Yes, these are the usual rules about the relationship between mutexs and condition variables. >> > There is sometimes a definite need for >1 condition variable associated with a single >> > mutex. The other way around might be definable, but I think would create a big mess. >> > >> > Our Thread interface documents only that Wait(m,c) must have m locked at the time of >> > call (and will return with it locked too, but not necessarily for the duration. >> > >> > Chapter 4 of SPwM3, "An Introduction to Programming with Threads" informally defines >> > a _monitor_ as "some data, a mutex, and zero or more condition variables" and says >> > further "A particular condition variable is always used in conjunction with the same >> > mutex and its data". (SPwM3, 4.3.2, p93) >> > >> > It looks to me like the formal specification of Wait, Signal, and Broadcast in >> > SRwM3, 5.3.3, p124 would allow a condition to simultaneously contain waiters >> > that passed in different mutexes. Each thread remembers locally what mutex it >> > wants to reacquire when it wakes up. >> > >> > I'm not sure we really want to commit to that. Client code's doing it would no doubt >> > create even trickier code than this kind of stuff already is. Off hand, I have a >> > hard time thinking of a reasonable use-case. >> > >> > If we decide to go with the informal statement, the implementations really should >> > enforce it by storing the associated mutex in the condition variable, and verifying >> > that additional waiters supply the same one. >> > >> > Probably, in most implementations, you could get away with moving the association >> > of a condition variable from one mutex to another when no threads were waiting on >> > the condition. Buy\t the only possible benefit I can think of would be memory savings, >> > not needing as many condition variables, since the different mutex associations would >> > be, well, mutually exclusive. >> > >> > >> > >> > >> > > i.e. while the lock used with a condition variable can change through time, >> > > there can be only one at a time. >> > > >> > > And then, presumably win32 condition variables and posix condition variables >> > > and Modula-3 Thread.Condition are all kind of similar. >> > > >> > > Hm, ok, a problem with merging the locks, is it leaves no lock for Signal/Broadcast. >> > > Perhaps they should use atomic/Interlocked. >> > > >> > > - Jay >> > > >> > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > -----! >> > --- >> > > From: jay.krell at cornell.edu >> > > To: rodney.m.bates at acm.org >> > > CC: m3devel at elegosoft.com >> > > Subject: RE: [M3devel] purported condition variable problems on Win32? >> > > Date: Thu, 21 Jul 2016 06:43:16 +0000 >> > > >> > > That question was meant for the other bug. >> > > >> > > Can this one be fixed by rechecking if tickets is positive before decrementing it? >> > > This seems like a big problem in Schmidt's code. >> > > >> > > I'm nervous about merging the locks, but maybe. >> > > In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. >> > > Though I realize that is actually necessarily part of what gets done. >> > > Schmidt clearly did not merge the locks, and the Java version does. >> > > The Java version also has its own simple and not-terrible critical section. >> > > The Java version also disappeared in newer releases and I didn't track down what happened to it. >> > > >> > > - Jay >> > > >> > > >> > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > -----! >> > --- >> > > From: jay.krell at cornell.edu >> > > To: rodney.m.bates at acm.org >> > > CC: m3devel at elegosoft.com >> > > Subject: RE: [M3devel] purported condition variable problems on Win32? >> > > Date: Thu, 21 Jul 2016 06:35:41 +0000 >> > > >> > > Am I understanding this? >> > > There are some waiters. >> > > There is a broadcast ("wake all"). Some waiters are woken. >> > > There are more waiters. >> > > There is a signal ("wake one"). >> > > There is a chance that some of the original waiters will remain waiting while the later waiters are not. >> > > That is, the signaled threads kind of steal the wake intended by the broadcast. >> > > >> > > >> > > Is this unfairness or incorrectness? >> > > It feels like unfairness. >> > > It feels like condition variables are specified fairly weakly. >> > > >> > > >> > > - Jay >> > > >> > > >> > > > Date: Wed, 20 Jul 2016 11:43:27 -0500 >> > > > From: rodney_bates at lcwb.coop >> > > > To: jay.krell at cornell.edu; rodney.m.bates at acm.org >> > > > CC: m3devel at elegosoft.com >> > > > Subject: Re: [M3devel] purported condition variable problems on Win32? >> > > > >> > > > In looking at ThreadWin32.m3 more, I think I see a significant bug. >> > > > >> > > > This gets confusing, since there are Modula-3 waits, signals, >> > > > etc. and similar concepts in Win32, but which are not the >> > > > same. I am going the try to put "M3-" and "Win-" prefixes on >> > > > things to make this clearer. I cartainly need to do this >> > > > for my own benefit, if for nobody else. >> > > > >> > > > Suppose two threads are both M3-waiting in Wait(m,c,...), further >> > > > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. >> > > > c.tickets = 0, and c.waiters = 2. >> > > > >> > > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), >> > > > which will allow both the waiters to Win-wakeup sometime soon. >> > > > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly >> > > > to eventually allow only one waiter to M3-wakeup) and increments >> > > > c.counter, setting things up so that the waitDone expression at :308 >> > > > will be TRUE for the waiting threads, the next time they evaluate it. >> > > > >> > > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, >> > > > releases c.lock, and tries to acquire m. The problem arises >> > > > because the second waiter can get to these steps before the first >> > > > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The >> > > > second to reach :308 will find c.tickets # 0, as did the first, >> > > > and proceed. >> > > > >> > > > Each will eventually, do a M3-wake. If this were Posix condition >> > > > variables, this would be wrong, because Posix says only one >> > > > waiter proceeds. For M3, it's OK, if unnecessary, because M3 >> > > > says one or more waiters proceed. However, each will, in the >> > > > process, get to :322, and DEC(c.tickets). Signal only provided >> > > > one ticket, but the waiters have taken two. c.tickets goes >> > > > negative. >> > > > >> > > > After this, when some thread can do another (M3-)Wait, and some >> > > > other does another (M3-)Signal, c.tickets increments only back to >> > > > zero, which means the new waiter will not M3-wake, even though it >> > > > should. >> > > > >> > > > >> > > > I am going to try to construct a test case that will force >> > > > this scenario. >> > > > >> > > > See more below: >> > > > >> > > > >> > > > On 07/18/2016 08:53 PM, Jay wrote: >> > > > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. >> > > > > >> > > > > Otherwise previous version I think had a very large lock. >> > > > > >> > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. >> > > > > >> > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. >> > > > > >> > > > > I guess you are saying it would be unfair, which is ok. >> > > > > >> > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. >> > > > > >> > > > > I'll check on the unused event. >> > > > > >> > > > > - Jay >> > > > > >> > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> I spent some time looking over the signal part of ThreadWin32.m3. A few >> > > > >> comments: >> > > > >> >> > > > >> This looks like a very direct implementation of Schmidts' generation count >> > > > >> algorithm, with a number of important little details additionally taken care >> > > > >> of. >> > > > >> >> > > > >> On small difference is at line 308 (B, below) where Schmidt does >> > > > >> c.counter>count instead of #. Usually, this would make no difference, >> > > > >> since counter only increases, but on a long-running program, it could >> > > > >> overflow. If this happened,as it often does, as silent wrap-around to >> > > > >> FIRST(INTEGER), some threads could be trapped for an extremely long >> > > > >> time, waiting for counter come back around greater than their count. As >> > > > >> coded in ThreadWin32, they would be allowed to proceed normally. >> > > > >> >> > > > >> The only glitch would be that, if there were simultaneously waiting threads >> > > > >> separated by NUMBER(INTEGER) truly different generations (but the same value >> > > > >> of c.counter), the recent ones could unfairly proceed along with the >> > > > >> extremely old ones. That seems extremely less likely than the >> > > > >> mere existence of an overflow. >> > > > >> >> > > > >> Both type Condition and type Activation have a field waitEvent: HANDLE. >> > > > >> Condition.waitEvent is extensively used, but Activation.waitEvent is >> > > > >> only created at :537 and deleted at :554, but never used in any real >> > > > >> way. It could be deleted. >> > > > >> >> > > > >> As for this code: >> > > > >> >> > > > >> EnterCriticalSection(conditionLock); >> > > > >> >> > > > >> (* Capture the value of the counter before we start waiting. >> > > > >> * We will not stop waiting until the counter changes. >> > > > >> * That is, we will not stop waiting until a signal >> > > > >> * comes in after we start waiting. >> > > > >> *) >> > > > >> >> > > > >> count := c.counter; >> > > > >> INC(c.waiters); >> > > > >> >> > > > >> LeaveCriticalSection(conditionLock); >> > > > >> m.release(); (* can this be moved to before taking conditionLock? *) >> > > > >> >> > > > >> No, it is important to sample and save c.counter in the order threads wait. >> > > > >> Retaining m until this is done ensures this. Otherwise, some other thread >> > > > >> that waited later could have altered c.counter before this one gets conditionLock >> > > > >> and saves its copy of c.counter. >> > > > >> >> > > > >> I have made several comment changes to ThreadWin32.m3 that would have made >> > > > >> it easier to vet. I will commit these, but only comments. It is either >> > > > >> very difficult or impossible for me to test or even recompile this module, >> > > > >> and, especially as fragile as this kind of code is, I don't want to commit >> > > > >> any substantive changes untested and uncompiled. >> > > > >> >> > > > >> More below: >> > > > >> >> > > > >>> On 07/07/2016 04:24 AM, Jay K wrote: >> > > > >>> So...I do NOT understand all of this. >> > > > >>> >> > > > >>> >> > > > >>> However, comparing the three implementations >> > > > >>> and attempting to understand ours, >> > > > >>> I am struck by the following >> > > > >>> >> > > > >>> - Our broadcast seems ok, but >> > > > >>> - our signal seems to wake everyone, >> > > > >>> and then...they race a bit, >> > > > >>> one will decide it is last, and >> > > > >>> reset the event, but every prior waiter >> > > > >>> is still woken. >> > > > >>> Why not merge the following chunks: >> > > > >> >> > > > >> I don't think this will work. It is important that, once a thread has decided, >> > > > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed >> > > > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, >> > > > >> if it is also the last of its generation to proceed, resets c.waitevent. This >> > > > >> is ensured by its having already reacquired m first, preventing any other thread >> > > > >> from returning from its Wait yet. >> > > > >> > > > I was wrong about this. There is already no fairness being enforced here. Even with >> > > > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is >> > > > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. >> > > > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters >> > > > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. >> > > > >> > > > Merging, as you proposed, is probably close to the fix for this bug. >> > > > >> > > > >> > > > >> >> > > > >> And it can't attempt to acquire m while already holding conditionLock, because this would >> > > > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while >> > > > >> already holding m, at line 265. >> > > > >> >> > > > >>> >> > > > >>> >> > > > >>> WHILE (NOT alerted) AND (NOT waitDone) DO >> > > > >>> . >> > > > >>> . >> > > > >>> . >> > > > >>> 1 >> > > > >>> EnterCriticalSection(conditionLock); >> > > > >>> waitDone := (c.tickets # 0 AND c.counter # count); >> > > > >> A:---------^ >> > > > >> B:--------------------------------------------------^ >> > > > >>> LeaveCriticalSection(conditionLock); >> > > > >>> END; (* WHILE *) >> > > > >>> IF waitDone THEN >> > > > >>> alerted := FALSE; >> > > > >>> END; >> > > > >>> m.acquire(); >> > > > >>> 2 >> > > > >>> EnterCriticalSection(conditionLock); >> > > > >>> DEC(c.waiters); >> > > > >>> IF waitDone THEN esp. here. >> > > > >>> DEC(c.tickets); >> > > > >> >> > > > >> C:---------------^ >> > > > >>> lastWaiter := (c.tickets = 0); >> > > > >>> END; >> > > > >>> LeaveCriticalSection(conditionLock); >> > > > >>> >> > > > >>> >> > > > >>> That is, if we decrement tickets earlier >> > > > >>> within the first critical section, >> > > > >>> while we will still wake everyone, >> > > > >>> only one will decide waitDone and the rest will keep looping. >> > > > >>> A downside of this, perhaps, is that waking all for Broadcast >> > > > >>> might be a little slower. >> > > > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads >> > > > >>> implementations, they seem to deal with this differently than us, >> > > > >>> and they each do about the same thing -- they use a counted semaphore. >> > > > >>> In the boost case, it appears they duplicate the semaphore for every >> > > > >>> notification generation, which seems expensive. >> > > > >>> >> > > > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. >> > > > >>> Or they can do their own reference counting? >> > > > >>> jdk7 seems to looke like jdk6. >> > > > >>> The code is gone in jdk8 and I can't easily find the delete in history. >> > > > >>> The lock merging jdk does probably helps here too. >> > > > >>> In fact they merge #1 and #2 above as a result. >> > > > >>> But they still initially wake all threads for signal, not just broadcast. >> > > > >>> There is also the problem that all the event waits >> > > > >>> are followed by EnterCriticalSection (or jdk facsimile). >> > > > >>> - Jay >> > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > ----! >> > -----! >> > > > --! >> > > > >> --- >> > > > >>> From: jay.krell at cornell.edu >> > > > >>> To: m3devel at elegosoft.com >> > > > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? >> > > > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 >> > > > >>> >> > > > >>> Here is another implementation to consider: >> > > > >>> >> > > > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp >> > > > >>> >> > > > >>> - Jay >> > > > >>> >> > > > >>> >> > > > >>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > ----! >> > -----! >> > > > --! >> > > > >> --- >> > > > >>> From: jay.krell at cornell.edu >> > > > >>> To: m3devel at elegosoft.com >> > > > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 >> > > > >>> Subject: [M3devel] purported condition variable problems on Win32? >> > > > >>> >> > > > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV >> > > > >>> >> > > > >>> vs. >> > > > >>> >> > > > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >> > > > >>> vs. >> > > > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain >> > > > >>> I wrote this: >> > > > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; >> > > > >>> alertable: BOOLEAN) RAISES {Alerted} = >> > > > >>> (* LL = m on entry and exit, but not for the duration >> > > > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp >> > > > >>> * NOTE that they merge the user lock and the condition lock. >> > > > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >> > > > >>> * "3.3. The Generation Count Solution" >> > > > >>> *) >> > > > >>> >> > > > >>> Do we have the problems described in README.CV? >> > > > >>> >> > > > >>> I haven't looked through the ACE code to see >> > > > >>> to what extent they resemble solution 3.3, or if they >> > > > >>> changed as a result of this discussion -- which I admit I don't understand >> > > > >>> and haven't read closely. >> > > > >>> >> > > > >>> >> > > > >>> Spurious wakeups are ok, though should be minimized. >> > > > >>> >> > > > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. >> > > > >>> >> > > > >>> Thank you, >> > > > >>> - Jay >> > > > >>> >> > > > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> > > > >>> >> > > > >>> >> > > > >>> _______________________________________________ >> > > > >>> M3devel mailing list >> > > > >>> M3devel at elegosoft.com >> > > > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> > > > >> >> > > > >> -- >> > > > >> Rodney Bates >> > > > >> rodney.m.bates at acm.org >> > > > >> >> > > > >> >> > > > > >> > > > >> > > > -- >> > > > Rodney Bates >> > > > rodney.m.bates at acm.org >> > >> > -- >> > Rodney Bates >> > rodney.m.bates at acm.org > > -- > Rodney Bates > rodney.m.bates at acm.org From jay.krell at cornell.edu Mon Jul 25 07:50:56 2016 From: jay.krell at cornell.edu (Jay K) Date: Mon, 25 Jul 2016 05:50:56 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> <578FAA2F.6020105@lcwb.coop>, , , <5793DB37.10603@lcwb.coop>, <5794ECAE.5070308@lcwb.coop>, Message-ID: btw, I think you are right. In particular, regarding the "directed notify" pattern,and the fact that we can have a per-thread event. I think you don't see this excact implementation stategy...you see people creating an event per wait...is becausewe have the feature and defect that we require ourown runtime to create our threads, and we don't interoperatewell with threads created otherwise...and there is a fix for this though,at least on Windows (where we are discussing anyway) viam3core.dll's DllMain, which we don't implement but probably should. So, let's go ahead and put support in Thread.T. or Thread.Activationand solve this? - Jay > From: jay.krell at cornell.edu > CC: jay.krell at cornell.edu; m3devel at elegosoft.com > Subject: Re: [M3devel] purported condition variable problems on Win32? > Date: Sun, 24 Jul 2016 14:53:12 -0700 > To: rodney.m.bates at acm.org > > It is a problem for merging mutex with condition lock, or rather, for having Wait rely on the external lock. > > - Jay > > > On Jul 24, 2016, at 9:28 AM, "Rodney M. Bates" wrote: > > > > No, they don't. But I don't think that presents a problem. > > > >> On 07/23/2016 06:37 PM, Jay K wrote: > >> I don't think Signal/Broadcast require caller be holding any lock, is a problem. > >> > >> - Jay > >> > >> > >> > >> > Date: Sat, 23 Jul 2016 16:01:43 -0500 > >> > From: rodney_bates at lcwb.coop > >> > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > >> > CC: m3devel at elegosoft.com > >> > Subject: Re: [M3devel] purported condition variable problems on Win32? > >> > > >> > > >> > > >> > On 07/22/2016 03:20 AM, Jay K wrote: > >> > > Here is very weak circumstantial evidence that the locks can be merged: > >> > > > >> > > > >> > > > >> > > https://msdn.microsoft.com/en-us/library/windows/desktop/ms682052(v=vs.85).aspx > >> > > > >> > > "It is often convenient to use more than one condition variable with the same lock. > >> > > For example, an implementation of a reader/writer lock might use a single critical > >> > > section but separate condition variables for readers and writers. > >> > > " > >> > > > >> > > > >> > > i.e. the mapping isn't one lock to one condition, but the convenient > >> > > construct is multiple conditions to one lock, not multiple locks to one condition. > >> > > > >> > > > >> > > http://linux.die.net/man/3/pthread_cond_wait > >> > > > >> > > " > >> > > The effect of using more than one mutex for concurrent pthread_cond_timedwait() or > >> > > pthread_cond_wait() operations on the same condition variable is undefined; that is, > >> > > a condition variable becomes bound to a unique mutex when a thread waits on the condition > >> > > variable, and this (dynamic) binding shall end when the wait returns. > >> > > " > >> > > > >> > > >> > Yes, these are the usual rules about the relationship between mutexs and condition variables. > >> > There is sometimes a definite need for >1 condition variable associated with a single > >> > mutex. The other way around might be definable, but I think would create a big mess. > >> > > >> > Our Thread interface documents only that Wait(m,c) must have m locked at the time of > >> > call (and will return with it locked too, but not necessarily for the duration. > >> > > >> > Chapter 4 of SPwM3, "An Introduction to Programming with Threads" informally defines > >> > a _monitor_ as "some data, a mutex, and zero or more condition variables" and says > >> > further "A particular condition variable is always used in conjunction with the same > >> > mutex and its data". (SPwM3, 4.3.2, p93) > >> > > >> > It looks to me like the formal specification of Wait, Signal, and Broadcast in > >> > SRwM3, 5.3.3, p124 would allow a condition to simultaneously contain waiters > >> > that passed in different mutexes. Each thread remembers locally what mutex it > >> > wants to reacquire when it wakes up. > >> > > >> > I'm not sure we really want to commit to that. Client code's doing it would no doubt > >> > create even trickier code than this kind of stuff already is. Off hand, I have a > >> > hard time thinking of a reasonable use-case. > >> > > >> > If we decide to go with the informal statement, the implementations really should > >> > enforce it by storing the associated mutex in the condition variable, and verifying > >> > that additional waiters supply the same one. > >> > > >> > Probably, in most implementations, you could get away with moving the association > >> > of a condition variable from one mutex to another when no threads were waiting on > >> > the condition. Buy\t the only possible benefit I can think of would be memory savings, > >> > not needing as many condition variables, since the different mutex associations would > >> > be, well, mutually exclusive. > >> > > >> > > >> > > >> > > >> > > i.e. while the lock used with a condition variable can change through time, > >> > > there can be only one at a time. > >> > > > >> > > And then, presumably win32 condition variables and posix condition variables > >> > > and Modula-3 Thread.Condition are all kind of similar. > >> > > > >> > > Hm, ok, a problem with merging the locks, is it leaves no lock for Signal/Broadcast. > >> > > Perhaps they should use atomic/Interlocked. > >> > > > >> > > - Jay > >> > > > >> > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > -----! > >> > --- > >> > > From: jay.krell at cornell.edu > >> > > To: rodney.m.bates at acm.org > >> > > CC: m3devel at elegosoft.com > >> > > Subject: RE: [M3devel] purported condition variable problems on Win32? > >> > > Date: Thu, 21 Jul 2016 06:43:16 +0000 > >> > > > >> > > That question was meant for the other bug. > >> > > > >> > > Can this one be fixed by rechecking if tickets is positive before decrementing it? > >> > > This seems like a big problem in Schmidt's code. > >> > > > >> > > I'm nervous about merging the locks, but maybe. > >> > > In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. > >> > > Though I realize that is actually necessarily part of what gets done. > >> > > Schmidt clearly did not merge the locks, and the Java version does. > >> > > The Java version also has its own simple and not-terrible critical section. > >> > > The Java version also disappeared in newer releases and I didn't track down what happened to it. > >> > > > >> > > - Jay > >> > > > >> > > > >> > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > -----! > >> > --- > >> > > From: jay.krell at cornell.edu > >> > > To: rodney.m.bates at acm.org > >> > > CC: m3devel at elegosoft.com > >> > > Subject: RE: [M3devel] purported condition variable problems on Win32? > >> > > Date: Thu, 21 Jul 2016 06:35:41 +0000 > >> > > > >> > > Am I understanding this? > >> > > There are some waiters. > >> > > There is a broadcast ("wake all"). Some waiters are woken. > >> > > There are more waiters. > >> > > There is a signal ("wake one"). > >> > > There is a chance that some of the original waiters will remain waiting while the later waiters are not. > >> > > That is, the signaled threads kind of steal the wake intended by the broadcast. > >> > > > >> > > > >> > > Is this unfairness or incorrectness? > >> > > It feels like unfairness. > >> > > It feels like condition variables are specified fairly weakly. > >> > > > >> > > > >> > > - Jay > >> > > > >> > > > >> > > > Date: Wed, 20 Jul 2016 11:43:27 -0500 > >> > > > From: rodney_bates at lcwb.coop > >> > > > To: jay.krell at cornell.edu; rodney.m.bates at acm.org > >> > > > CC: m3devel at elegosoft.com > >> > > > Subject: Re: [M3devel] purported condition variable problems on Win32? > >> > > > > >> > > > In looking at ThreadWin32.m3 more, I think I see a significant bug. > >> > > > > >> > > > This gets confusing, since there are Modula-3 waits, signals, > >> > > > etc. and similar concepts in Win32, but which are not the > >> > > > same. I am going the try to put "M3-" and "Win-" prefixes on > >> > > > things to make this clearer. I cartainly need to do this > >> > > > for my own benefit, if for nobody else. > >> > > > > >> > > > Suppose two threads are both M3-waiting in Wait(m,c,...), further > >> > > > Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. > >> > > > c.tickets = 0, and c.waiters = 2. > >> > > > > >> > > > Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), > >> > > > which will allow both the waiters to Win-wakeup sometime soon. > >> > > > Before releasing c.lock, Signal increments c.tickets to 1 (supposedly > >> > > > to eventually allow only one waiter to M3-wakeup) and increments > >> > > > c.counter, setting things up so that the waitDone expression at :308 > >> > > > will be TRUE for the waiting threads, the next time they evaluate it. > >> > > > > >> > > > Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, > >> > > > releases c.lock, and tries to acquire m. The problem arises > >> > > > because the second waiter can get to these steps before the first > >> > > > can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The > >> > > > second to reach :308 will find c.tickets # 0, as did the first, > >> > > > and proceed. > >> > > > > >> > > > Each will eventually, do a M3-wake. If this were Posix condition > >> > > > variables, this would be wrong, because Posix says only one > >> > > > waiter proceeds. For M3, it's OK, if unnecessary, because M3 > >> > > > says one or more waiters proceed. However, each will, in the > >> > > > process, get to :322, and DEC(c.tickets). Signal only provided > >> > > > one ticket, but the waiters have taken two. c.tickets goes > >> > > > negative. > >> > > > > >> > > > After this, when some thread can do another (M3-)Wait, and some > >> > > > other does another (M3-)Signal, c.tickets increments only back to > >> > > > zero, which means the new waiter will not M3-wake, even though it > >> > > > should. > >> > > > > >> > > > > >> > > > I am going to try to construct a test case that will force > >> > > > this scenario. > >> > > > > >> > > > See more below: > >> > > > > >> > > > > >> > > > On 07/18/2016 08:53 PM, Jay wrote: > >> > > > > I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. > >> > > > > > >> > > > > Otherwise previous version I think had a very large lock. > >> > > > > > >> > > > > Changing the < to !=, is that really ok? I get that 32bit overflow is possible. > >> > > > > > >> > > > > We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. > >> > > > > > >> > > > > I guess you are saying it would be unfair, which is ok. > >> > > > > > >> > > > > There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. > >> > > > > > >> > > > > I'll check on the unused event. > >> > > > > > >> > > > > - Jay > >> > > > > > >> > > > >> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > >> I spent some time looking over the signal part of ThreadWin32.m3. A few > >> > > > >> comments: > >> > > > >> > >> > > > >> This looks like a very direct implementation of Schmidts' generation count > >> > > > >> algorithm, with a number of important little details additionally taken care > >> > > > >> of. > >> > > > >> > >> > > > >> On small difference is at line 308 (B, below) where Schmidt does > >> > > > >> c.counter>count instead of #. Usually, this would make no difference, > >> > > > >> since counter only increases, but on a long-running program, it could > >> > > > >> overflow. If this happened,as it often does, as silent wrap-around to > >> > > > >> FIRST(INTEGER), some threads could be trapped for an extremely long > >> > > > >> time, waiting for counter come back around greater than their count. As > >> > > > >> coded in ThreadWin32, they would be allowed to proceed normally. > >> > > > >> > >> > > > >> The only glitch would be that, if there were simultaneously waiting threads > >> > > > >> separated by NUMBER(INTEGER) truly different generations (but the same value > >> > > > >> of c.counter), the recent ones could unfairly proceed along with the > >> > > > >> extremely old ones. That seems extremely less likely than the > >> > > > >> mere existence of an overflow. > >> > > > >> > >> > > > >> Both type Condition and type Activation have a field waitEvent: HANDLE. > >> > > > >> Condition.waitEvent is extensively used, but Activation.waitEvent is > >> > > > >> only created at :537 and deleted at :554, but never used in any real > >> > > > >> way. It could be deleted. > >> > > > >> > >> > > > >> As for this code: > >> > > > >> > >> > > > >> EnterCriticalSection(conditionLock); > >> > > > >> > >> > > > >> (* Capture the value of the counter before we start waiting. > >> > > > >> * We will not stop waiting until the counter changes. > >> > > > >> * That is, we will not stop waiting until a signal > >> > > > >> * comes in after we start waiting. > >> > > > >> *) > >> > > > >> > >> > > > >> count := c.counter; > >> > > > >> INC(c.waiters); > >> > > > >> > >> > > > >> LeaveCriticalSection(conditionLock); > >> > > > >> m.release(); (* can this be moved to before taking conditionLock? *) > >> > > > >> > >> > > > >> No, it is important to sample and save c.counter in the order threads wait. > >> > > > >> Retaining m until this is done ensures this. Otherwise, some other thread > >> > > > >> that waited later could have altered c.counter before this one gets conditionLock > >> > > > >> and saves its copy of c.counter. > >> > > > >> > >> > > > >> I have made several comment changes to ThreadWin32.m3 that would have made > >> > > > >> it easier to vet. I will commit these, but only comments. It is either > >> > > > >> very difficult or impossible for me to test or even recompile this module, > >> > > > >> and, especially as fragile as this kind of code is, I don't want to commit > >> > > > >> any substantive changes untested and uncompiled. > >> > > > >> > >> > > > >> More below: > >> > > > >> > >> > > > >>> On 07/07/2016 04:24 AM, Jay K wrote: > >> > > > >>> So...I do NOT understand all of this. > >> > > > >>> > >> > > > >>> > >> > > > >>> However, comparing the three implementations > >> > > > >>> and attempting to understand ours, > >> > > > >>> I am struck by the following > >> > > > >>> > >> > > > >>> - Our broadcast seems ok, but > >> > > > >>> - our signal seems to wake everyone, > >> > > > >>> and then...they race a bit, > >> > > > >>> one will decide it is last, and > >> > > > >>> reset the event, but every prior waiter > >> > > > >>> is still woken. > >> > > > >>> Why not merge the following chunks: > >> > > > >> > >> > > > >> I don't think this will work. It is important that, once a thread has decided, > >> > > > >> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed > >> > > > >> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, > >> > > > >> if it is also the last of its generation to proceed, resets c.waitevent. This > >> > > > >> is ensured by its having already reacquired m first, preventing any other thread > >> > > > >> from returning from its Wait yet. > >> > > > > >> > > > I was wrong about this. There is already no fairness being enforced here. Even with > >> > > > only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is > >> > > > arbitrary anyway. There is no benefit in trying to control the order they reacquire m. > >> > > > c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters > >> > > > that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. > >> > > > > >> > > > Merging, as you proposed, is probably close to the fix for this bug. > >> > > > > >> > > > > >> > > > >> > >> > > > >> And it can't attempt to acquire m while already holding conditionLock, because this would > >> > > > >> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while > >> > > > >> already holding m, at line 265. > >> > > > >> > >> > > > >>> > >> > > > >>> > >> > > > >>> WHILE (NOT alerted) AND (NOT waitDone) DO > >> > > > >>> . > >> > > > >>> . > >> > > > >>> . > >> > > > >>> 1 > >> > > > >>> EnterCriticalSection(conditionLock); > >> > > > >>> waitDone := (c.tickets # 0 AND c.counter # count); > >> > > > >> A:---------^ > >> > > > >> B:--------------------------------------------------^ > >> > > > >>> LeaveCriticalSection(conditionLock); > >> > > > >>> END; (* WHILE *) > >> > > > >>> IF waitDone THEN > >> > > > >>> alerted := FALSE; > >> > > > >>> END; > >> > > > >>> m.acquire(); > >> > > > >>> 2 > >> > > > >>> EnterCriticalSection(conditionLock); > >> > > > >>> DEC(c.waiters); > >> > > > >>> IF waitDone THEN esp. here. > >> > > > >>> DEC(c.tickets); > >> > > > >> > >> > > > >> C:---------------^ > >> > > > >>> lastWaiter := (c.tickets = 0); > >> > > > >>> END; > >> > > > >>> LeaveCriticalSection(conditionLock); > >> > > > >>> > >> > > > >>> > >> > > > >>> That is, if we decrement tickets earlier > >> > > > >>> within the first critical section, > >> > > > >>> while we will still wake everyone, > >> > > > >>> only one will decide waitDone and the rest will keep looping. > >> > > > >>> A downside of this, perhaps, is that waking all for Broadcast > >> > > > >>> might be a little slower. > >> > > > >>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads > >> > > > >>> implementations, they seem to deal with this differently than us, > >> > > > >>> and they each do about the same thing -- they use a counted semaphore. > >> > > > >>> In the boost case, it appears they duplicate the semaphore for every > >> > > > >>> notification generation, which seems expensive. > >> > > > >>> > >> > > > >>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. > >> > > > >>> Or they can do their own reference counting? > >> > > > >>> jdk7 seems to looke like jdk6. > >> > > > >>> The code is gone in jdk8 and I can't easily find the delete in history. > >> > > > >>> The lock merging jdk does probably helps here too. > >> > > > >>> In fact they merge #1 and #2 above as a result. > >> > > > >>> But they still initially wake all threads for signal, not just broadcast. > >> > > > >>> There is also the problem that all the event waits > >> > > > >>> are followed by EnterCriticalSection (or jdk facsimile). > >> > > > >>> - Jay > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > ----! > >> > -----! > >> > > > --! > >> > > > >> --- > >> > > > >>> From: jay.krell at cornell.edu > >> > > > >>> To: m3devel at elegosoft.com > >> > > > >>> Subject: RE: [M3devel] purported condition variable problems on Win32? > >> > > > >>> Date: Tue, 5 Jul 2016 08:26:36 +0000 > >> > > > >>> > >> > > > >>> Here is another implementation to consider: > >> > > > >>> > >> > > > >>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp > >> > > > >>> > >> > > > >>> - Jay > >> > > > >>> > >> > > > >>> > >> > > > >>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > > ----! > >> > -----! > >> > > > --! > >> > > > >> --- > >> > > > >>> From: jay.krell at cornell.edu > >> > > > >>> To: m3devel at elegosoft.com > >> > > > >>> Date: Tue, 5 Jul 2016 08:19:23 +0000 > >> > > > >>> Subject: [M3devel] purported condition variable problems on Win32? > >> > > > >>> > >> > > > >>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV > >> > > > >>> > >> > > > >>> vs. > >> > > > >>> > >> > > > >>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >> > > > >>> vs. > >> > > > >>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain > >> > > > >>> I wrote this: > >> > > > >>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; > >> > > > >>> alertable: BOOLEAN) RAISES {Alerted} = > >> > > > >>> (* LL = m on entry and exit, but not for the duration > >> > > > >>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp > >> > > > >>> * NOTE that they merge the user lock and the condition lock. > >> > > > >>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html > >> > > > >>> * "3.3. The Generation Count Solution" > >> > > > >>> *) > >> > > > >>> > >> > > > >>> Do we have the problems described in README.CV? > >> > > > >>> > >> > > > >>> I haven't looked through the ACE code to see > >> > > > >>> to what extent they resemble solution 3.3, or if they > >> > > > >>> changed as a result of this discussion -- which I admit I don't understand > >> > > > >>> and haven't read closely. > >> > > > >>> > >> > > > >>> > >> > > > >>> Spurious wakeups are ok, though should be minimized. > >> > > > >>> > >> > > > >>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. > >> > > > >>> > >> > > > >>> Thank you, > >> > > > >>> - Jay > >> > > > >>> > >> > > > >>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >> > > > >>> > >> > > > >>> > >> > > > >>> _______________________________________________ > >> > > > >>> M3devel mailing list > >> > > > >>> M3devel at elegosoft.com > >> > > > >>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel > >> > > > >> > >> > > > >> -- > >> > > > >> Rodney Bates > >> > > > >> rodney.m.bates at acm.org > >> > > > >> > >> > > > >> > >> > > > > > >> > > > > >> > > > -- > >> > > > Rodney Bates > >> > > > rodney.m.bates at acm.org > >> > > >> > -- > >> > Rodney Bates > >> > rodney.m.bates at acm.org > > > > -- > > Rodney Bates > > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Jul 25 09:27:49 2016 From: jay.krell at cornell.edu (Jay K) Date: Mon, 25 Jul 2016 07:27:49 -0000 Subject: [M3devel] Grisu vs. NT386 Message-ID: Fyi, Grisu reveals at least two problems in the NT386 backend.In particular, it uses LONGINT, and we don't have much code that does. The two problem are that NT386: - generates function calls for e.g. 64bit integer muliplication - doesn't like generating function calls within function calls, i.e. VAR a,b: LONGINT; Foo(a * b); It fails an assert -- when it goes to do the multiply, which is a function call, it asserts that no function call is in progress. For now I'll change Grisu: VAR a,b,t: LONGINT; t := a * b; Foo(t); but fixing NT386 shouldn't be difficult. Really, we should allow for: Foo(Bar(a * b) * b); .. the thing is, the frontend handles this -- it produces temporaries for function return results: t1 := Bar(a * b); Foo(t1 * a * b); but it doesn't do so for things like add/subtract/multiply/divide. Perhaps it should.Decent backends, including possibly NT386, will enregister the temporaries and generate good code anyway. Thoughts? The alternative is NT386 needs to maintain a stack where it doesn't previously -- a stack of pending function calls. 2) Grisu uses loophole between LONGREAL and LONGINT. This is reasonable, but perhaps not previously seen.NT386 requires loopholes that convert between integers and floats to be using REAL, at least for one direction.This seem merely historical and it should allow LONGREAL, or in particular, it should require no size change.Historically NT386 backend maintained an internal stack, of 32bit sized things -- variables and constants.I extended that to be variably sized, i.e. 32bits or 64bits, but this code wasn't updated. It should be trivial, maybe just relaxing the assertion. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Mon Jul 25 18:18:29 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 25 Jul 2016 16:18:29 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop> <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> <578FAA2F.6020105@lcwb.coop> <5793DB37.10603@lcwb.coop> <5794ECAE.5070308@lcwb.coop> Message-ID: <57963BB2.3000602@lcwb.coop> On 07/24/2016 04:53 PM, Jay wrote: > It is a problem for merging mutex with condition lock, or rather, for having Wait rely on the external lock. > I guess I don't understand what you mean by this. We don't want the condition variable and the mutex to be the same object, because it can be desirable to have more than one condition (in the informal sense) that different threads waiting on the same mutex-protected data are waiting for. The classic example is a FIFO message buffer with limited capacity. Some threads want to receive a message and wait for one to be available. Others want to send, and wait for the buffer to have space for it. So you want one Condition variable for each condition and signallers Signal the one they just made TRUE. You could do this with only one condition variable that gets a Broadcast when "something has changed", with all waiters waking up and rechecking for what they want, but it's overly awkward and entails a whole lot of unnecessary wake/sleep looping. "lock" is pretty ambiguous here. Could at least mean M3-Mutex, M3-Condition-variable, Win-critical-region, Win-event. > - Jay > >> On Jul 24, 2016, at 9:28 AM, "Rodney M. Bates" wrote: >> >> No, they don't. But I don't think that presents a problem. >> >>> On 07/23/2016 06:37 PM, Jay K wrote: >>> I don't think Signal/Broadcast require caller be holding any lock, is a problem. >>> >>> - Jay >>> >>> >>> >>>> Date: Sat, 23 Jul 2016 16:01:43 -0500 >>>> From: rodney_bates at lcwb.coop >>>> To: jay.krell at cornell.edu; rodney.m.bates at acm.org >>>> CC: m3devel at elegosoft.com >>>> Subject: Re: [M3devel] purported condition variable problems on Win32? >>>> >>>> >>>> >>>> On 07/22/2016 03:20 AM, Jay K wrote: >>>>> Here is very weak circumstantial evidence that the locks can be merged: >>>>> >>>>> >>>>> >>>>> https://msdn.microsoft.com/en-us/library/windows/desktop/ms682052(v=vs.85).aspx >>>>> >>>>> "It is often convenient to use more than one condition variable with the same lock. >>>>> For example, an implementation of a reader/writer lock might use a single critical >>>>> section but separate condition variables for readers and writers. >>>>> " >>>>> >>>>> >>>>> i.e. the mapping isn't one lock to one condition, but the convenient >>>>> construct is multiple conditions to one lock, not multiple locks to one condition. >>>>> >>>>> >>>>> http://linux.die.net/man/3/pthread_cond_wait >>>>> >>>>> " >>>>> The effect of using more than one mutex for concurrent pthread_cond_timedwait() or >>>>> pthread_cond_wait() operations on the same condition variable is undefined; that is, >>>>> a condition variable becomes bound to a unique mutex when a thread waits on the condition >>>>> variable, and this (dynamic) binding shall end when the wait returns. >>>>> " >>>>> >>>> >>>> Yes, these are the usual rules about the relationship between mutexs and condition variables. >>>> There is sometimes a definite need for >1 condition variable associated with a single >>>> mutex. The other way around might be definable, but I think would create a big mess. >>>> >>>> Our Thread interface documents only that Wait(m,c) must have m locked at the time of >>>> call (and will return with it locked too, but not necessarily for the duration. >>>> >>>> Chapter 4 of SPwM3, "An Introduction to Programming with Threads" informally defines >>>> a _monitor_ as "some data, a mutex, and zero or more condition variables" and says >>>> further "A particular condition variable is always used in conjunction with the same >>>> mutex and its data". (SPwM3, 4.3.2, p93) >>>> >>>> It looks to me like the formal specification of Wait, Signal, and Broadcast in >>>> SRwM3, 5.3.3, p124 would allow a condition to simultaneously contain waiters >>>> that passed in different mutexes. Each thread remembers locally what mutex it >>>> wants to reacquire when it wakes up. >>>> >>>> I'm not sure we really want to commit to that. Client code's doing it would no doubt >>>> create even trickier code than this kind of stuff already is. Off hand, I have a >>>> hard time thinking of a reasonable use-case. >>>> >>>> If we decide to go with the informal statement, the implementations really should >>>> enforce it by storing the associated mutex in the condition variable, and verifying >>>> that additional waiters supply the same one. >>>> >>>> Probably, in most implementations, you could get away with moving the association >>>> of a condition variable from one mutex to another when no threads were waiting on >>>> the condition. Buy\t the only possible benefit I can think of would be memory savings, >>>> not needing as many condition variables, since the different mutex associations would >>>> be, well, mutually exclusive. >>>> >>>> >>>> >>>> >>>>> i.e. while the lock used with a condition variable can change through time, >>>>> there can be only one at a time. >>>>> >>>>> And then, presumably win32 condition variables and posix condition variables >>>>> and Modula-3 Thread.Condition are all kind of similar. >>>>> >>>>> Hm, ok, a problem with merging the locks, is it leaves no lock for Signal/Broadcast. >>>>> Perhaps they should use atomic/Interlocked. >>>>> >>>>> - Jay >>>>> >>>>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> -----! >>>> --- >>>>> From: jay.krell at cornell.edu >>>>> To: rodney.m.bates at acm.org >>>>> CC: m3devel at elegosoft.com >>>>> Subject: RE: [M3devel] purported condition variable problems on Win32? >>>>> Date: Thu, 21 Jul 2016 06:43:16 +0000 >>>>> >>>>> That question was meant for the other bug. >>>>> >>>>> Can this one be fixed by rechecking if tickets is positive before decrementing it? >>>>> This seems like a big problem in Schmidt's code. >>>>> >>>>> I'm nervous about merging the locks, but maybe. >>>>> In particular, I'm the multiple release/acquires in service of the condition variable, also releasing/acquiring the user variable. >>>>> Though I realize that is actually necessarily part of what gets done. >>>>> Schmidt clearly did not merge the locks, and the Java version does. >>>>> The Java version also has its own simple and not-terrible critical section. >>>>> The Java version also disappeared in newer releases and I didn't track down what happened to it. >>>>> >>>>> - Jay >>>>> >>>>> >>>>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> -----! >>>> --- >>>>> From: jay.krell at cornell.edu >>>>> To: rodney.m.bates at acm.org >>>>> CC: m3devel at elegosoft.com >>>>> Subject: RE: [M3devel] purported condition variable problems on Win32? >>>>> Date: Thu, 21 Jul 2016 06:35:41 +0000 >>>>> >>>>> Am I understanding this? >>>>> There are some waiters. >>>>> There is a broadcast ("wake all"). Some waiters are woken. >>>>> There are more waiters. >>>>> There is a signal ("wake one"). >>>>> There is a chance that some of the original waiters will remain waiting while the later waiters are not. >>>>> That is, the signaled threads kind of steal the wake intended by the broadcast. >>>>> >>>>> >>>>> Is this unfairness or incorrectness? >>>>> It feels like unfairness. >>>>> It feels like condition variables are specified fairly weakly. >>>>> >>>>> >>>>> - Jay >>>>> >>>>> >>>>>> Date: Wed, 20 Jul 2016 11:43:27 -0500 >>>>>> From: rodney_bates at lcwb.coop >>>>>> To: jay.krell at cornell.edu; rodney.m.bates at acm.org >>>>>> CC: m3devel at elegosoft.com >>>>>> Subject: Re: [M3devel] purported condition variable problems on Win32? >>>>>> >>>>>> In looking at ThreadWin32.m3 more, I think I see a significant bug. >>>>>> >>>>>> This gets confusing, since there are Modula-3 waits, signals, >>>>>> etc. and similar concepts in Win32, but which are not the >>>>>> same. I am going the try to put "M3-" and "Win-" prefixes on >>>>>> things to make this clearer. I cartainly need to do this >>>>>> for my own benefit, if for nobody else. >>>>>> >>>>>> Suppose two threads are both M3-waiting in Wait(m,c,...), further >>>>>> Win-waiting on c.waitEvent in WaitForMultipleObjects at :301. >>>>>> c.tickets = 0, and c.waiters = 2. >>>>>> >>>>>> Now, (M3-)Signal (c) happens. This does (Win-)SetEvent(c.waitEvent), >>>>>> which will allow both the waiters to Win-wakeup sometime soon. >>>>>> Before releasing c.lock, Signal increments c.tickets to 1 (supposedly >>>>>> to eventually allow only one waiter to M3-wakeup) and increments >>>>>> c.counter, setting things up so that the waitDone expression at :308 >>>>>> will be TRUE for the waiting threads, the next time they evaluate it. >>>>>> >>>>>> Each waiter in turn Win-wakes, gets c.lock, sets waitDone TRUE, >>>>>> releases c.lock, and tries to acquire m. The problem arises >>>>>> because the second waiter can get to these steps before the first >>>>>> can acquire m, acquire c.lock, and DEC(c.tickets) at :322. The >>>>>> second to reach :308 will find c.tickets # 0, as did the first, >>>>>> and proceed. >>>>>> >>>>>> Each will eventually, do a M3-wake. If this were Posix condition >>>>>> variables, this would be wrong, because Posix says only one >>>>>> waiter proceeds. For M3, it's OK, if unnecessary, because M3 >>>>>> says one or more waiters proceed. However, each will, in the >>>>>> process, get to :322, and DEC(c.tickets). Signal only provided >>>>>> one ticket, but the waiters have taken two. c.tickets goes >>>>>> negative. >>>>>> >>>>>> After this, when some thread can do another (M3-)Wait, and some >>>>>> other does another (M3-)Signal, c.tickets increments only back to >>>>>> zero, which means the new waiter will not M3-wake, even though it >>>>>> should. >>>>>> >>>>>> >>>>>> I am going to try to construct a test case that will force >>>>>> this scenario. >>>>>> >>>>>> See more below: >>>>>> >>>>>> >>>>>> On 07/18/2016 08:53 PM, Jay wrote: >>>>>>> I wrote this version. Right, based on Schmidt. I also found similar code in Java. There are comments as to both of these in the code. >>>>>>> >>>>>>> Otherwise previous version I think had a very large lock. >>>>>>> >>>>>>> Changing the < to !=, is that really ok? I get that 32bit overflow is possible. >>>>>>> >>>>>>> We can change to a 64bit counter if < is required. That really can't rollover. I.e. Even a 64bit cycle counter can't rollover, and this would advance much slower. >>>>>>> >>>>>>> I guess you are saying it would be unfair, which is ok. >>>>>>> >>>>>>> There is pattern called "directed notification" or such that might be preferable? It has a certain significant inefficiency though, like creating an event per notify. I think it is what boost's condition variable are using. >>>>>>> >>>>>>> I'll check on the unused event. >>>>>>> >>>>>>> - Jay >>>>>>> >>>>>>>> On Jul 18, 2016, at 7:25 AM, "Rodney M. Bates" wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I spent some time looking over the signal part of ThreadWin32.m3. A few >>>>>>>> comments: >>>>>>>> >>>>>>>> This looks like a very direct implementation of Schmidts' generation count >>>>>>>> algorithm, with a number of important little details additionally taken care >>>>>>>> of. >>>>>>>> >>>>>>>> On small difference is at line 308 (B, below) where Schmidt does >>>>>>>> c.counter>count instead of #. Usually, this would make no difference, >>>>>>>> since counter only increases, but on a long-running program, it could >>>>>>>> overflow. If this happened,as it often does, as silent wrap-around to >>>>>>>> FIRST(INTEGER), some threads could be trapped for an extremely long >>>>>>>> time, waiting for counter come back around greater than their count. As >>>>>>>> coded in ThreadWin32, they would be allowed to proceed normally. >>>>>>>> >>>>>>>> The only glitch would be that, if there were simultaneously waiting threads >>>>>>>> separated by NUMBER(INTEGER) truly different generations (but the same value >>>>>>>> of c.counter), the recent ones could unfairly proceed along with the >>>>>>>> extremely old ones. That seems extremely less likely than the >>>>>>>> mere existence of an overflow. >>>>>>>> >>>>>>>> Both type Condition and type Activation have a field waitEvent: HANDLE. >>>>>>>> Condition.waitEvent is extensively used, but Activation.waitEvent is >>>>>>>> only created at :537 and deleted at :554, but never used in any real >>>>>>>> way. It could be deleted. >>>>>>>> >>>>>>>> As for this code: >>>>>>>> >>>>>>>> EnterCriticalSection(conditionLock); >>>>>>>> >>>>>>>> (* Capture the value of the counter before we start waiting. >>>>>>>> * We will not stop waiting until the counter changes. >>>>>>>> * That is, we will not stop waiting until a signal >>>>>>>> * comes in after we start waiting. >>>>>>>> *) >>>>>>>> >>>>>>>> count := c.counter; >>>>>>>> INC(c.waiters); >>>>>>>> >>>>>>>> LeaveCriticalSection(conditionLock); >>>>>>>> m.release(); (* can this be moved to before taking conditionLock? *) >>>>>>>> >>>>>>>> No, it is important to sample and save c.counter in the order threads wait. >>>>>>>> Retaining m until this is done ensures this. Otherwise, some other thread >>>>>>>> that waited later could have altered c.counter before this one gets conditionLock >>>>>>>> and saves its copy of c.counter. >>>>>>>> >>>>>>>> I have made several comment changes to ThreadWin32.m3 that would have made >>>>>>>> it easier to vet. I will commit these, but only comments. It is either >>>>>>>> very difficult or impossible for me to test or even recompile this module, >>>>>>>> and, especially as fragile as this kind of code is, I don't want to commit >>>>>>>> any substantive changes untested and uncompiled. >>>>>>>> >>>>>>>> More below: >>>>>>>> >>>>>>>>> On 07/07/2016 04:24 AM, Jay K wrote: >>>>>>>>> So...I do NOT understand all of this. >>>>>>>>> >>>>>>>>> >>>>>>>>> However, comparing the three implementations >>>>>>>>> and attempting to understand ours, >>>>>>>>> I am struck by the following >>>>>>>>> >>>>>>>>> - Our broadcast seems ok, but >>>>>>>>> - our signal seems to wake everyone, >>>>>>>>> and then...they race a bit, >>>>>>>>> one will decide it is last, and >>>>>>>>> reset the event, but every prior waiter >>>>>>>>> is still woken. >>>>>>>>> Why not merge the following chunks: >>>>>>>> >>>>>>>> I don't think this will work. It is important that, once a thread has decided, >>>>>>>> at A, that it can fairly proceed, it is guaranteed to be the next thread to proceed >>>>>>>> (i.e, to return from [Alert]Wait), before it decrements c.tickets at C, and, thus, >>>>>>>> if it is also the last of its generation to proceed, resets c.waitevent. This >>>>>>>> is ensured by its having already reacquired m first, preventing any other thread >>>>>>>> from returning from its Wait yet. >>>>>> >>>>>> I was wrong about this. There is already no fairness being enforced here. Even with >>>>>> only one ticked, all waiters Win-wake at :301, the order they get c.lock at :307 is >>>>>> arbitrary anyway. There is no benefit in trying to control the order they reacquire m. >>>>>> c.counter has nothing to do with fairness. What it does is prevent Win-awakened waiters >>>>>> that waited after the most recent Signal/Broadcast from M3-waking, a correctness issue. >>>>>> >>>>>> Merging, as you proposed, is probably close to the fix for this bug. >>>>>> >>>>>> >>>>>>>> >>>>>>>> And it can't attempt to acquire m while already holding conditionLock, because this would >>>>>>>> invite deadlock, since, elsewhere, a thread can try to acquire conditionLock while >>>>>>>> already holding m, at line 265. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> WHILE (NOT alerted) AND (NOT waitDone) DO >>>>>>>>> . >>>>>>>>> . >>>>>>>>> . >>>>>>>>> 1 >>>>>>>>> EnterCriticalSection(conditionLock); >>>>>>>>> waitDone := (c.tickets # 0 AND c.counter # count); >>>>>>>> A:---------^ >>>>>>>> B:--------------------------------------------------^ >>>>>>>>> LeaveCriticalSection(conditionLock); >>>>>>>>> END; (* WHILE *) >>>>>>>>> IF waitDone THEN >>>>>>>>> alerted := FALSE; >>>>>>>>> END; >>>>>>>>> m.acquire(); >>>>>>>>> 2 >>>>>>>>> EnterCriticalSection(conditionLock); >>>>>>>>> DEC(c.waiters); >>>>>>>>> IF waitDone THEN esp. here. >>>>>>>>> DEC(c.tickets); >>>>>>>> >>>>>>>> C:---------------^ >>>>>>>>> lastWaiter := (c.tickets = 0); >>>>>>>>> END; >>>>>>>>> LeaveCriticalSection(conditionLock); >>>>>>>>> >>>>>>>>> >>>>>>>>> That is, if we decrement tickets earlier >>>>>>>>> within the first critical section, >>>>>>>>> while we will still wake everyone, >>>>>>>>> only one will decide waitDone and the rest will keep looping. >>>>>>>>> A downside of this, perhaps, is that waking all for Broadcast >>>>>>>>> might be a little slower. >>>>>>>>> but more so, in both the "ptw32" (pthreads for win32) and Boost threads >>>>>>>>> implementations, they seem to deal with this differently than us, >>>>>>>>> and they each do about the same thing -- they use a counted semaphore. >>>>>>>>> In the boost case, it appears they duplicate the semaphore for every >>>>>>>>> notification generation, which seems expensive. >>>>>>>>> >>>>>>>>> Perhaps if they can guarantee some lifetimes, they don't need to duplicate it. >>>>>>>>> Or they can do their own reference counting? >>>>>>>>> jdk7 seems to looke like jdk6. >>>>>>>>> The code is gone in jdk8 and I can't easily find the delete in history. >>>>>>>>> The lock merging jdk does probably helps here too. >>>>>>>>> In fact they merge #1 and #2 above as a result. >>>>>>>>> But they still initially wake all threads for signal, not just broadcast. >>>>>>>>> There is also the problem that all the event waits >>>>>>>>> are followed by EnterCriticalSection (or jdk facsimile). >>>>>>>>> - Jay >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> ----! >>>> -----! >>>>>> --! >>>>>>>> --- >>>>>>>>> From: jay.krell at cornell.edu >>>>>>>>> To: m3devel at elegosoft.com >>>>>>>>> Subject: RE: [M3devel] purported condition variable problems on Win32? >>>>>>>>> Date: Tue, 5 Jul 2016 08:26:36 +0000 >>>>>>>>> >>>>>>>>> Here is another implementation to consider: >>>>>>>>> >>>>>>>>> https://github.com/boostorg/thread/blob/develop/include/boost/thread/win32/condition_variable.hpp >>>>>>>>> >>>>>>>>> - Jay >>>>>>>>> >>>>>>>>> >>>>>>>>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> ----! >>>> -----! >>>>>> --! >>>>>>>> --- >>>>>>>>> From: jay.krell at cornell.edu >>>>>>>>> To: m3devel at elegosoft.com >>>>>>>>> Date: Tue, 5 Jul 2016 08:19:23 +0000 >>>>>>>>> Subject: [M3devel] purported condition variable problems on Win32? >>>>>>>>> >>>>>>>>> https://sourceforge.net/p/pthreads4w/code/ci/master/tree/README.CV >>>>>>>>> >>>>>>>>> vs. >>>>>>>>> >>>>>>>>> http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >>>>>>>>> vs. >>>>>>>>> https://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32.m3?rev=1.210.2.1;content-type=text%2Fplain >>>>>>>>> I wrote this: >>>>>>>>> PROCEDURE XWait(m: Mutex; c: Condition; act: Activation; >>>>>>>>> alertable: BOOLEAN) RAISES {Alerted} = >>>>>>>>> (* LL = m on entry and exit, but not for the duration >>>>>>>>> * see C:\src\jdk-6u14-ea-src-b05-jrl-23_apr_2009\hotspot\agent\src\os\win32\Monitor.cpp >>>>>>>>> * NOTE that they merge the user lock and the condition lock. >>>>>>>>> * http://www.cs.wustl.edu/~schmidt/win32-cv-1.html >>>>>>>>> * "3.3. The Generation Count Solution" >>>>>>>>> *) >>>>>>>>> >>>>>>>>> Do we have the problems described in README.CV? >>>>>>>>> >>>>>>>>> I haven't looked through the ACE code to see >>>>>>>>> to what extent they resemble solution 3.3, or if they >>>>>>>>> changed as a result of this discussion -- which I admit I don't understand >>>>>>>>> and haven't read closely. >>>>>>>>> >>>>>>>>> >>>>>>>>> Spurious wakeups are ok, though should be minimized. >>>>>>>>> >>>>>>>>> I'd still rather not drop pre-Vista support but I realize it becomes more interesting as time advances. >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> - Jay >>>>>>>>> >>>>>>>>> _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> M3devel mailing list >>>>>>>>> M3devel at elegosoft.com >>>>>>>>> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >>>>>>>> >>>>>>>> -- >>>>>>>> Rodney Bates >>>>>>>> rodney.m.bates at acm.org >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Rodney Bates >>>>>> rodney.m.bates at acm.org >>>> >>>> -- >>>> Rodney Bates >>>> rodney.m.bates at acm.org >> >> -- >> Rodney Bates >> rodney.m.bates at acm.org > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Mon Jul 25 18:34:24 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 25 Jul 2016 16:34:24 -0000 Subject: [M3devel] purported condition variable problems on Win32? In-Reply-To: References: <578BF33B.2030605@lcwb.coop> <578CE6C7.2010007@lcwb.coop>, <595D45C5-8280-4B50-BE75-EA974957B838@gmail.com> <578FAA2F.6020105@lcwb.coop>, , , <5793DB37.10603@lcwb.coop>, <5794ECAE.5070308@lcwb.coop>, Message-ID: <57963F79.8000906@lcwb.coop> BTW, I see yet a third concurrency bug in ThreadWin32.m3; In AssignSlot, the code to allocate the first slots array, starting at :465 (* make sure we have room to register this guy *) IF slots = NIL THEN LeaveCriticalSection(ADR(slotLock)); slots := NEW (REF ARRAY OF T, 20); EnterCriticalSection(ADR(slotLock)); END; has a race with itself, that needs to be handled the same way that the immediately following code, which enlarges an already-existing slots array, does. (* make sure we have room to register this guy *) IF slots = NIL THEN LeaveCriticalSection(ADR(slotLock)); new_slots := NEW (REF ARRAY OF T, 20); EnterCriticalSection(ADR(slotLock)); IF slots = NIL (* still. *) THEN (* We won any races. *) slots := new_slots; END; END; Otherwise, not only could a different thread have allocated slots after the NIL check and before the assignment to it, it could have put something into its slots, and even enlarged it. Also, both the initial and the enlarging NEW for slots should have an explicit NIL check for out of memory. On 07/25/2016 12:50 AM, Jay K wrote: > btw, I think you are right. > > > In particular, regarding the "directed notify" pattern, > and the fact that we can have a per-thread event. > > > I think you don't see this excact implementation stategy... > you see people creating an event per wait...is because > we have the feature and defect that we require our > own runtime to create our threads, and we don't interoperate > well with threads created otherwise...and there is a fix for this though, > at least on Windows (where we are discussing anyway) via > m3core.dll's DllMain, which we don't implement but probably should. > > > So, let's go ahead and put support in Thread.T. or Thread.Activation > and solve this? > > - Jay > > > -- Rodney Bates rodney.m.bates at acm.org From jay.krell at cornell.edu Tue Jul 26 08:42:31 2016 From: jay.krell at cornell.edu (Jay K) Date: Tue, 26 Jul 2016 06:42:31 -0000 Subject: [M3devel] Grisu vs. NT386 In-Reply-To: References: Message-ID: I think I can fix the nested helper call easily enough in the backend. But note that NT386 is definitely currently broken since the introduction of Grisu. We might also consider having the frontend always form parameters into temporaries. Not clear if it should do this as needed for a few known cases, like LONGINT multiplication, or for all integer and float operations on all types, despite most of them not requiring function calls. Also not clear wrt longer standing helper functions like div/mod. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: Grisu vs. NT386 Date: Mon, 25 Jul 2016 07:27:46 +0000 Fyi, Grisu reveals at least two problems in the NT386 backend.In particular, it uses LONGINT, and we don't have much code that does. The two problem are that NT386: - generates function calls for e.g. 64bit integer muliplication - doesn't like generating function calls within function calls, i.e. VAR a,b: LONGINT; Foo(a * b); It fails an assert -- when it goes to do the multiply, which is a function call, it asserts that no function call is in progress. For now I'll change Grisu: VAR a,b,t: LONGINT; t := a * b; Foo(t); but fixing NT386 shouldn't be difficult. Really, we should allow for: Foo(Bar(a * b) * b); .. the thing is, the frontend handles this -- it produces temporaries for function return results: t1 := Bar(a * b); Foo(t1 * a * b); but it doesn't do so for things like add/subtract/multiply/divide. Perhaps it should.Decent backends, including possibly NT386, will enregister the temporaries and generate good code anyway. Thoughts? The alternative is NT386 needs to maintain a stack where it doesn't previously -- a stack of pending function calls. 2) Grisu uses loophole between LONGREAL and LONGINT. This is reasonable, but perhaps not previously seen.NT386 requires loopholes that convert between integers and floats to be using REAL, at least for one direction.This seem merely historical and it should allow LONGREAL, or in particular, it should require no size change.Historically NT386 backend maintained an internal stack, of 32bit sized things -- variables and constants.I extended that to be variably sized, i.e. 32bits or 64bits, but this code wasn't updated. It should be trivial, maybe just relaxing the assertion. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Jul 28 09:11:07 2016 From: jay.krell at cornell.edu (Jay K) Date: Thu, 28 Jul 2016 07:11:07 -0000 Subject: [M3devel] Grisu vs. NT386 In-Reply-To: References: , Message-ID: This should all be fixed now in the NT386 backend. Verification welcome. Work could be shifted to the frontend for some of this, perhaps. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: RE: Grisu vs. NT386 Date: Tue, 26 Jul 2016 06:42:28 +0000 I think I can fix the nested helper call easily enough in the backend. But note that NT386 is definitely currently broken since the introduction of Grisu. We might also consider having the frontend always form parameters into temporaries. Not clear if it should do this as needed for a few known cases, like LONGINT multiplication, or for all integer and float operations on all types, despite most of them not requiring function calls. Also not clear wrt longer standing helper functions like div/mod. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: Grisu vs. NT386 Date: Mon, 25 Jul 2016 07:27:46 +0000 Fyi, Grisu reveals at least two problems in the NT386 backend.In particular, it uses LONGINT, and we don't have much code that does. The two problem are that NT386: - generates function calls for e.g. 64bit integer muliplication - doesn't like generating function calls within function calls, i.e. VAR a,b: LONGINT; Foo(a * b); It fails an assert -- when it goes to do the multiply, which is a function call, it asserts that no function call is in progress. For now I'll change Grisu: VAR a,b,t: LONGINT; t := a * b; Foo(t); but fixing NT386 shouldn't be difficult. Really, we should allow for: Foo(Bar(a * b) * b); .. the thing is, the frontend handles this -- it produces temporaries for function return results: t1 := Bar(a * b); Foo(t1 * a * b); but it doesn't do so for things like add/subtract/multiply/divide. Perhaps it should.Decent backends, including possibly NT386, will enregister the temporaries and generate good code anyway. Thoughts? The alternative is NT386 needs to maintain a stack where it doesn't previously -- a stack of pending function calls. 2) Grisu uses loophole between LONGREAL and LONGINT. This is reasonable, but perhaps not previously seen.NT386 requires loopholes that convert between integers and floats to be using REAL, at least for one direction.This seem merely historical and it should allow LONGREAL, or in particular, it should require no size change.Historically NT386 backend maintained an internal stack, of 32bit sized things -- variables and constants.I extended that to be variably sized, i.e. 32bits or 64bits, but this code wasn't updated. It should be trivial, maybe just relaxing the assertion. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Thu Jul 28 22:02:10 2016 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Thu, 28 Jul 2016 20:02:10 -0000 Subject: [M3devel] A win32 thread test case Message-ID: <579A646D.3020404@lcwb.coop> I have attached a test case program designed to expose one of the bugs I think I see in ThreadWin32.m3. It runs correctly on AMD64_LINUX. I expect it to fail on Windows, but don't have a Windows machine I can try it on. Anybody willing to try it? It's a self-contained Main module. Just compile and run it. It will announce what it finds. -- Rodney Bates rodney.m.bates at acm.org -------------- next part -------------- (* Test to expose a bug, called the "Twice used ticket" bug, in ThreadWin32.m3: Multiple waiters take multiple tickets when only one was provided by Signal. The result is that the tickets count of a Condition variable goes negative, causing later lost wakeups in Signal. *) MODULE Main ; IMPORT Fmt ; IMPORT Stdio ; IMPORT Thread ; IMPORT Wr ; PROCEDURE W ( Msg : TEXT ) = <* FATAL Thread . Alerted *> <* FATAL Wr . Failure *> BEGIN Wr . PutText ( Stdio . stdout , Msg ) ; Wr . PutText ( Stdio . stdout , Wr . EOL ) ; Wr . Flush ( Stdio . stdout ) END W ; TYPE ThreadNo = [ 0 .. 4 ] ; CONST ThreadNoNull = 0 ; PROCEDURE ThImage ( ThN : ThreadNo ) : TEXT = BEGIN RETURN "Thread " & Fmt . Int ( ThN ) END ThImage (* We want to deterministically control the sequence of thread operations, to create a test case. It is hard to do this without using Condition variables, but we are testing Condition variables, so can't really assume they work correctly. We resort to using a hodge-podge of MUTEXs, busy waiting, and pauses, hoping they are long enough. *) ; CONST BusyWaitInterval = 0.1D0 ; CONST QuiesceInterval = 3.0D0 ; TYPE State = { Null , Idle , Acq , Rel , Wait , Wait2 , Sig } ; TYPE StateSet = SET OF State ; VAR StateMutex : MUTEX ; VAR States := ARRAY ThreadNo OF State { State . Null , .. } (* LL = StateMutex *) ; VAR Holder : ThreadNo := ThreadNoNull (* LL = StateMutex *) ; PROCEDURE StateImage ( St : State ) : TEXT = VAR LResult : TEXT ; BEGIN CASE St OF State . Null => LResult := "null" | State . Idle => LResult := "idle" | State . Acq => LResult := "entering Acquire" | State . Rel => LResult := "entering Release" | State . Wait => LResult := "entering Wait" | State . Wait2 => LResult := "asleep in Wait" | State . Sig => LResult := "entering Signal" END (* CASE *) ; RETURN LResult END StateImage ; EXCEPTION Failure ; VAR ActionMutex : MUTEX ; TYPE ActionProc = PROCEDURE ( ThN : ThreadNo ) ; VAR ActionProcs := ARRAY ThreadNo OF ActionProc { NIL , .. } (* LL = ActionProc *) ; PROCEDURE Action ( ThN : ThreadNo ; Apply : ActionProc ) (* Ask thread number ThN to execute Apply. *) = BEGIN LOCK ActionMutex DO WITH WAct = ActionProcs [ ThN ] DO <* ASSERT WAct = NIL *> WAct := Apply END END ; LOOP (* Wait for test thread to take the direction. *) Thread . Pause ( BusyWaitInterval ) ; LOCK ActionMutex DO IF ActionProcs [ ThN ] = NIL THEN EXIT END (* IF *) END (* LOCK *) END (* LOOP *) END Action ; PROCEDURE ActionWait ( ThN : ThreadNo ; Apply : ActionProc ) RAISES { Failure } (* Ask thread number ThN to execute Apply, expecting Apply to do a Wait(TestMutex, TestCondition), which further implies ThN already holds TestMutex. Note that it waited by its release of TextMutex. *) = BEGIN Action ( ThN , Apply ) ; LOOP (* Wait for ThN to release TestMutex. No other thread is contending for TestMutex, so this is an indication that ThN has stopped inside Wait. *) LOCK TestMutex DO LOCK StateMutex DO WITH WSt = States [ ThN ] DO IF WSt # State . Wait THEN W ( ThImage ( ThN ) & " Failed to wait in Wait." ) ; RAISE Failure ELSE <* ASSERT Holder = ThN *> Holder := ThreadNoNull ; WSt := State . Wait2 ; W ( ThImage ( ThN ) & " Waiting." ) ; EXIT END (* IF *) END (* WITH *) END (* LOCK StateMutex *) END (* LOCK TestMutex*) END (* LOOP *) END ActionWait ; PROCEDURE WaitForHeld ( ) : ThreadNo (* Busy wait until some thread holds TestMutex, then return its ThreadNo. *) = VAR LHolder : ThreadNo ; BEGIN LOOP Thread . Pause ( BusyWaitInterval ) ; LOCK StateMutex DO LHolder := Holder END (* LOCK *) ; IF LHolder # ThreadNoNull THEN EXIT END END (* LOOP *) ; RETURN LHolder END WaitForHeld ; PROCEDURE WaitForStateSet ( ThN : ThreadNo ; Sts : StateSet ) (* Busy wait for thread number ThN to get into one of Sts. *) = BEGIN LOOP LOCK StateMutex DO WITH WSt = States [ ThN ] DO IF WSt IN Sts THEN (* This is what we want. *) EXIT END (* IF *) END (* WITH *) END (* LOCK *) ; Thread . Pause ( BusyWaitInterval ) END (* LOOP *) END WaitForStateSet ; PROCEDURE WaitForState ( ThN : ThreadNo ; St : State ) (* Busy wait for action thread to get into St. *) = BEGIN WaitForStateSet ( ThN , StateSet { St } ) END WaitForState ; PROCEDURE NoteWhetherStateSet ( ThN : ThreadNo ; Sts : StateSet ; YesMsg , NoMsg := "" ) : BOOLEAN (* Write whether thread number ThN is in one of Sts and return it. *) = VAR LResult : BOOLEAN ; VAR LState : State ; VAR LMsg : TEXT ; BEGIN LOCK StateMutex DO LState := States [ ThN ] ; LResult := LState IN Sts ; LMsg := ThImage ( ThN ) & " is " & StateImage ( LState ) ; IF LResult THEN IF YesMsg # NIL THEN LMsg := LMsg & YesMsg END ELSE IF NoMsg # NIL THEN LMsg := LMsg & NoMsg END END (* IF *) ; W ( LMsg ) END (* LOCK *) ; RETURN LResult END NoteWhetherStateSet ; PROCEDURE NoteWhetherState ( ThN : ThreadNo ; St : State ; YesMsg , NoMsg := "" ) : BOOLEAN (* Write whether thread number ThN is in St and return it. *) = BEGIN RETURN NoteWhetherStateSet ( ThN , StateSet { St } , YesMsg , NoMsg ) END NoteWhetherState ; VAR TestMutex : MUTEX ; VAR TestCond : Thread . Condition ; VAR Threads : ARRAY ThreadNo OF Thread . T ; TYPE Cl = Thread . Closure OBJECT ClThN : ThreadNo OVERRIDES apply := TestApply END ; VAR Closures : ARRAY ThreadNo OF Cl ; PROCEDURE TestApply ( Self : Cl ) : REFANY (* Loop for threads that execute test actions on Mutexs and Conditions. *) = VAR LProc : ActionProc ; BEGIN LOOP (* Busy wait for work. *) Thread . Pause ( 0.2D0 (* Not too extremely busy. *) + FLOAT ( Self . ClThN , LONGREAL ) * 0.01D0 (* ^Avoid staying in phase. *) ) ; LOCK ActionMutex DO WITH WProc = ActionProcs [ Self . ClThN ] DO IF WProc # NIL THEN (* We have some work to do. *) LProc := WProc ; WProc := NIL END (* IF *) END (* WITH *) END (* LOCK *) ; IF LProc # NIL THEN LProc ( Self . ClThN ) (* ^May or may not return right away. *) ; LProc := NIL END (* IF *) END (* LOOP *) ; <* NOWARN *> (* Intentionally unreachable. *) RETURN NIL END TestApply ; PROCEDURE DoAcq ( ThN : ThreadNo ) = BEGIN LOCK StateMutex DO States [ ThN ] := State . Acq END ; W ( ThImage ( ThN ) & " Entering Acquire of TestMutex." ) ; Thread . Acquire ( TestMutex ) ; W ( ThImage ( ThN ) & " Acquired TestMutex." ) ; LOCK StateMutex DO WITH WThN = States [ ThN ] DO <* ASSERT WThN = State . Acq *> WThN := State . Idle ; <* ASSERT Holder = ThreadNoNull *> Holder := ThN END (* WITH *) END (* LOCK *) END DoAcq ; PROCEDURE DoRel ( ThN : ThreadNo ) = BEGIN LOCK StateMutex DO States [ ThN ] := State . Rel END ; W ( ThImage ( ThN ) & " Entering release of TestMutex." ) ; Thread . Release ( TestMutex ) ; W ( ThImage ( ThN ) & " Released TestMutex." ) ; LOCK StateMutex DO WITH WThN = States [ ThN ] DO <* ASSERT WThN = State . Rel *> WThN := State . Idle ; <* ASSERT Holder = ThN *> Holder := ThreadNoNull END (* WITH *) END (* LOCK *) END DoRel ; PROCEDURE DoWait ( ThN : ThreadNo ) = BEGIN LOCK StateMutex DO WITH WThN = States [ ThN ] DO <* ASSERT Holder = ThN *> <* ASSERT WThN = State . Idle *> WThN := State . Wait END (* WITH *) END (* LOCK *) ; W ( ThImage ( ThN ) & " Entering Wait on TestCond." ) ; Thread . Wait ( TestMutex , TestCond ) ; W ( ThImage ( ThN ) & " Was Signalled in TestCond and reacquired TestMutex." ) ; LOCK StateMutex DO WITH WThN = States [ ThN ] DO <* ASSERT WThN = State . Wait2 *> States [ ThN ] := State . Idle ; Holder := ThN END (* WITH *) END (* LOCK *) END DoWait ; PROCEDURE DoSignal ( ThN : ThreadNo ) = BEGIN LOCK StateMutex DO States [ ThN ] := State . Sig END ; W ( ThImage ( ThN ) & " Entering Signal on TestCond" ) ; Thread . Signal ( TestCond ) ; W ( ThImage ( ThN ) & " TestCond Signalled " ) ; LOCK StateMutex DO States [ ThN ] := State . Idle END END DoSignal ; PROCEDURE ForceSignalled ( ThN : ThreadNo ) (* PRE: Only ThN could hold TestMutex. If thread ThN is asleep in TestCondition waiting for a Signal, Signal it and let it reacquire TestMutex. Either way, finally make it release TestMutex. *) = VAR LThN : ThreadNo ; BEGIN (* Allow ThN time to acquire TestMutex, if it has been signalled. *) Thread . Pause ( QuiesceInterval ) ; IF NoteWhetherState ( ThN , State . Wait2 ) THEN Action ( 3 , DoSignal ) ; WaitForState ( 3 , State . Idle ) ; LThN := WaitForHeld ( ) ; <* ASSERT LThN = ThN *> END (* IF *) ; Action ( ThN , DoRel ) ; WaitForState ( ThN , State . Idle ) END ForceSignalled ; PROCEDURE TestSeq ( ) = VAR LThNo : ThreadNo ; BEGIN TRY (* Thread 1 waits: *) Action ( 1 , DoAcq ) ; WaitForState ( 1 , State . Idle ) ; ActionWait ( 1 , DoWait ) ; WaitForState ( 1 , State . Wait2 ) (* Thread 2 waits: *) ; Action ( 2 , DoAcq ) ; WaitForState ( 2 , State . Idle ) ; ActionWait ( 2 , DoWait ) ; WaitForState ( 2 , State . Wait2 ) (* Here, there are two waiters. *) (* The Wait implementation must unblock at least one of the waiting threads, but is allowed to unblock up to all of them. The Posix and PThread implementations unblock only one. The buggy Win32 implementation unblocks both in unpredictable order. We want to get both unblocked from inside Wait and also let them acquire and release TestMutex, to set up consistently, in preparation for testing for the actual bug. *) (* Hold TestMutex to limit how far 1 and 2 can get after Wait unblocks them. *) ; Action ( 3 , DoAcq ) ; WaitForState ( 3 , State . Idle ) (* Signal. Each of 1 & 2, when unblocked, will block again trying to reacquire TestMutex.*) ; Action ( 3 , DoSignal ) ; WaitForState ( 3 , State . Idle ) (* Now release TestMutex, so the first thread unblocked inside Wait by the Signal will acquire it. *) ; Action ( 3 , DoRel ) ; WaitForState ( 3 , State . Idle ) (* See what thread was unblocked. *) ; LThNo := WaitForHeld ( ) ; IF LThNo = 1 THEN (* 1 holds TestMutex. 2 will be asleep, either in TestCondition or for TestMutex. *) Action ( 1 , DoRel ) ; WaitForState ( 1 , State . Idle ) ; ForceSignalled ( 2 ) ELSE (* 1 has not gotten out of Wait. 2 Must have done so. *) <* ASSERT LThNo = 2 *> Action ( 2 , DoRel ) ; WaitForState ( 2 , State . Idle ) ; ForceSignalled ( 1 ) END (* IF *) (* Tread 4 Waits. *) ; Action ( 4 , DoAcq ) ; WaitForState ( 4 , State . Idle ) ; ActionWait ( 4 , DoWait ) ; WaitForState ( 4 , State . Wait2 ) (* Thread 3 Signals. *) ; Action ( 3 , DoAcq ) ; WaitForState ( 3 , State . Idle ) ; Action ( 3 , DoSignal ) ; WaitForState ( 3 , State . Idle ) ; Action ( 3 , DoRel ) ; WaitForState ( 3 , State . Idle ) (* Give 4 time to reacquire TestMutex, if it was signalled *) ; Thread . Pause ( QuiesceInterval ) (* See whether 4 got signalled. *) ; WaitForStateSet ( 4 , StateSet { State . Wait2 , State . Idle } ) ; IF NoteWhetherState ( 4 , State . Idle ) THEN Action ( 4 , DoRel ) ; WaitForState ( 4 , State . Idle ) ; W ( "SUCCESS: all as expected.") ELSE W ( "FAILURE: This is the \"twice used ticket\" bug we are testing for." ) ; RAISE Failure END (* IF *) EXCEPT | Failure => END (* EXCEPT *) END TestSeq ; PROCEDURE Init ( ) = BEGIN ActionMutex := NEW ( MUTEX ) ; StateMutex := NEW ( MUTEX ) ; TestMutex := NEW ( MUTEX ) ; TestCond := NEW ( Thread . Condition ) ; FOR LThNo := FIRST ( ThreadNo ) TO LAST ( ThreadNo ) DO Closures [ LThNo ] := NEW ( Cl , ClThN := LThNo ) ; States [ LThNo ] := State . Idle (* Waiting for directions. *) ; Threads [ LThNo ] := Thread . Fork ( Closures [ LThNo ] ) END (* FOR *) END Init ; BEGIN Init ( ) ; TestSeq ( ) END Main . From jay.krell at cornell.edu Sat Jul 30 23:18:15 2016 From: jay.krell at cornell.edu (Jay K) Date: Sat, 30 Jul 2016 21:18:15 -0000 Subject: [M3devel] A win32 thread test case In-Reply-To: <579A646D.3020404@lcwb.coop> References: <579A646D.3020404@lcwb.coop> Message-ID: I fixed the new_slots problem. I agree there are problems here. I think Java got away with a similar implementation because their interface is a bit different. i.e. the lock and condition are merged, and/or notification requires the caller to take the lock. We can't do this. We allow signal/broadcast w/o holding a lock. We'd have to resort to a global lock I guess. I suspect nobody else uses this solution. I believe we can solve it better and worse than others. Most libraries do not have their own "CreateThread" function.That is, most libraries let you call pthread_create or Win32 CreateThread,and the library will interoperate with any thread that comes its way.And most libraries don't seem to use thread locals in their solution.By "most libraries", I mean the pthreads on win32 package and Boost. Modula-3 isn't clearly this way. This isn't great, but it is the currentsituation. I believe it is fixable.That is, Modula-3 I believe requires using its library to create threads.This is not great for interoperation. You want to be able to be usedby code that isn't away of you, that just creates threads in the "normal" wayfor their platform. And then, either the current state, or a fix I have in mind, can be takenadvantage of to do "directed notify" w/o creating a kernel eventper wait or notify, like other solutions do. As to the fix, to not require threads go through our "CreateThread",I believe on Windows (which is all that is relevant here), we should havea DllMain that allocates thread locals. We can only easily have a DllMain if we are a .dll.Or we can use __declspec(thread).__declspec(thread) works if the .exe is statically linked to the exe or the dll,or on Vista and newer, but not in a .dll loaded by LoadLibrary prior to Vista.Perhaps that is good enough. If we require Vista then we can just drop a bunch of our code and usethe Win32 condition variables presumably.I remain curious how the condition variables work there, i.e. we can'timplement them ourselves, but it is possible they either require kernelsupport or are well beyond our abilities. - Jay > Date: Thu, 28 Jul 2016 15:00:45 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: A win32 thread test case > > I have attached a test case program designed to expose one of the bugs > I think I see in ThreadWin32.m3. It runs correctly on AMD64_LINUX. > I expect it to fail on Windows, but don't have a Windows machine I > can try it on. > > Anybody willing to try it? It's a self-contained Main module. Just > compile and run it. It will announce what it finds. > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 00:15:28 2016 From: jay.krell at cornell.edu (Jay K) Date: Sat, 30 Jul 2016 22:15:28 -0000 Subject: [M3devel] why thread slots? Message-ID: What is the point of all this "slot" stuff in the thread library? It seems kind of self-referential and pointless. I might try removing it and see what the results are. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 02:47:27 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 00:47:27 -0000 Subject: [M3devel] A win32 thread test case In-Reply-To: References: <579A646D.3020404@lcwb.coop>, Message-ID: ThreadPThread.m3 had the same narrow startup race condition. I fixed it.Again I'm not seeing commit mails. On the larger matter: I think there is a solution, in that, if you look atThreadPThread.m3 and how it uses Posix condition variables,in that it does a fair amount of the work itself,you can term this "condition within condition". So I think maybe we can use the counting solutionand 1) merge the locks and/or 2) take the lock in Signal/Broadcast. But I have to look at it more. Specifically, in ThreadPThread.m3: Broadcast: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; next := t.nextWaiter; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Signal: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Becaise they signal while locked, we can implement this limitedform of condition variable, and then use the Modula-3 pthread logicusing that. Make some sense? Essentially we will have almost-posix condition variables,and exactly-posix mutexes, and then we can use just the pthread codeon top of that. Almost-posix condition variable is a posix condition variable, exceptit is guaranteed you will call pthread_cond_signal with the samelock held as when you call pthread_cond_wait. On a pthread system, the almost-posix condition variables and exactly-posixmutexes are just pass through to the underlying pthreads. On a win32 system, an almost-posix condition variable is similarto what we have now, except, because we know this lock is held,"XWait" can remove some of the leave/enter cycles and remove race conditions. Essentially we only need a "monitor", which is one lock and one condition variable. OR, we can use "directed notification" where each thread has an event, and threads are linked through a "wait block" i.e. the condition. This should work well too. It depends on us owing CreateThread, which we already do. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sat, 30 Jul 2016 21:18:08 +0000 I fixed the new_slots problem. I agree there are problems here. I think Java got away with a similar implementation because their interface is a bit different. i.e. the lock and condition are merged, and/or notification requires the caller to take the lock. We can't do this. We allow signal/broadcast w/o holding a lock. We'd have to resort to a global lock I guess. I suspect nobody else uses this solution. I believe we can solve it better and worse than others. Most libraries do not have their own "CreateThread" function.That is, most libraries let you call pthread_create or Win32 CreateThread,and the library will interoperate with any thread that comes its way.And most libraries don't seem to use thread locals in their solution.By "most libraries", I mean the pthreads on win32 package and Boost. Modula-3 isn't clearly this way. This isn't great, but it is the currentsituation. I believe it is fixable.That is, Modula-3 I believe requires using its library to create threads.This is not great for interoperation. You want to be able to be usedby code that isn't away of you, that just creates threads in the "normal" wayfor their platform. And then, either the current state, or a fix I have in mind, can be takenadvantage of to do "directed notify" w/o creating a kernel eventper wait or notify, like other solutions do. As to the fix, to not require threads go through our "CreateThread",I believe on Windows (which is all that is relevant here), we should havea DllMain that allocates thread locals. We can only easily have a DllMain if we are a .dll.Or we can use __declspec(thread).__declspec(thread) works if the .exe is statically linked to the exe or the dll,or on Vista and newer, but not in a .dll loaded by LoadLibrary prior to Vista.Perhaps that is good enough. If we require Vista then we can just drop a bunch of our code and usethe Win32 condition variables presumably.I remain curious how the condition variables work there, i.e. we can'timplement them ourselves, but it is possible they either require kernelsupport or are well beyond our abilities. - Jay > Date: Thu, 28 Jul 2016 15:00:45 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: A win32 thread test case > > I have attached a test case program designed to expose one of the bugs > I think I see in ThreadWin32.m3. It runs correctly on AMD64_LINUX. > I expect it to fail on Windows, but don't have a Windows machine I > can try it on. > > Anybody willing to try it? It's a self-contained Main module. Just > compile and run it. It will announce what it finds. > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 04:52:05 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 02:52:05 -0000 Subject: [M3devel] A win32 thread test case In-Reply-To: References: <579A646D.3020404@lcwb.coop>, , Message-ID: While I agree this is confusing, it is holding together for me and I have mostly coded it up. Repeating myself: In this scheme there are two types of condition variable. There are Modula-3 condition variables and posix-like/schmidt-like condition variables. The posix-like/smidt-like condition variables: Can be implemented directly via posix condition variables (if Windows pre-Vista had them). They are implemented as a slight modification of the Smidt counter method. The differences are that condition.lock is removed. Caller of cond_signal/cond_broadcast is responsible to hold the same lock as he holds when calling cond_wait. Some of the lock/unlock cycles are removed, and I hope therefore all the bugs are fixed. I still have to work through your examples. Modula-3 condition variables are then built on top of pthread-like/schmidt condition variables just like ThreadPThread.m3. This includes "alertable". The separate event previously in the Win32 implementation becomes a boolean plus cond_signal.XPause changes to call XWait, essentially. I was thinking we could reuse the Win32 thread alerting feature, but this alerts for other reasons so maybe isn't approprirate. If this works, we should be able to share more code between the pthread and win32 implementations.We just need a name for this slightly constrained pthread interface, i.e. where the condition variables assume the lock is held by the caller of signal/broadcast (we don't use broadcast). - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 00:47:24 +0000 ThreadPThread.m3 had the same narrow startup race condition. I fixed it.Again I'm not seeing commit mails. On the larger matter: I think there is a solution, in that, if you look atThreadPThread.m3 and how it uses Posix condition variables,in that it does a fair amount of the work itself,you can term this "condition within condition". So I think maybe we can use the counting solutionand 1) merge the locks and/or 2) take the lock in Signal/Broadcast. But I have to look at it more. Specifically, in ThreadPThread.m3: Broadcast: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; next := t.nextWaiter; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Signal: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Becaise they signal while locked, we can implement this limitedform of condition variable, and then use the Modula-3 pthread logicusing that. Make some sense? Essentially we will have almost-posix condition variables,and exactly-posix mutexes, and then we can use just the pthread codeon top of that. Almost-posix condition variable is a posix condition variable, exceptit is guaranteed you will call pthread_cond_signal with the samelock held as when you call pthread_cond_wait. On a pthread system, the almost-posix condition variables and exactly-posixmutexes are just pass through to the underlying pthreads. On a win32 system, an almost-posix condition variable is similarto what we have now, except, because we know this lock is held,"XWait" can remove some of the leave/enter cycles and remove race conditions. Essentially we only need a "monitor", which is one lock and one condition variable. OR, we can use "directed notification" where each thread has an event, and threads are linked through a "wait block" i.e. the condition. This should work well too. It depends on us owing CreateThread, which we already do. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sat, 30 Jul 2016 21:18:08 +0000 I fixed the new_slots problem. I agree there are problems here. I think Java got away with a similar implementation because their interface is a bit different. i.e. the lock and condition are merged, and/or notification requires the caller to take the lock. We can't do this. We allow signal/broadcast w/o holding a lock. We'd have to resort to a global lock I guess. I suspect nobody else uses this solution. I believe we can solve it better and worse than others. Most libraries do not have their own "CreateThread" function.That is, most libraries let you call pthread_create or Win32 CreateThread,and the library will interoperate with any thread that comes its way.And most libraries don't seem to use thread locals in their solution.By "most libraries", I mean the pthreads on win32 package and Boost. Modula-3 isn't clearly this way. This isn't great, but it is the currentsituation. I believe it is fixable.That is, Modula-3 I believe requires using its library to create threads.This is not great for interoperation. You want to be able to be usedby code that isn't away of you, that just creates threads in the "normal" wayfor their platform. And then, either the current state, or a fix I have in mind, can be takenadvantage of to do "directed notify" w/o creating a kernel eventper wait or notify, like other solutions do. As to the fix, to not require threads go through our "CreateThread",I believe on Windows (which is all that is relevant here), we should havea DllMain that allocates thread locals. We can only easily have a DllMain if we are a .dll.Or we can use __declspec(thread).__declspec(thread) works if the .exe is statically linked to the exe or the dll,or on Vista and newer, but not in a .dll loaded by LoadLibrary prior to Vista.Perhaps that is good enough. If we require Vista then we can just drop a bunch of our code and usethe Win32 condition variables presumably.I remain curious how the condition variables work there, i.e. we can'timplement them ourselves, but it is possible they either require kernelsupport or are well beyond our abilities. - Jay > Date: Thu, 28 Jul 2016 15:00:45 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: A win32 thread test case > > I have attached a test case program designed to expose one of the bugs > I think I see in ThreadWin32.m3. It runs correctly on AMD64_LINUX. > I expect it to fail on Windows, but don't have a Windows machine I > can try it on. > > Anybody willing to try it? It's a self-contained Main module. Just > compile and run it. It will announce what it finds. > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 05:54:58 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 03:54:58 -0000 Subject: [M3devel] joinMu global? Message-ID: Why is joinMu global, instead of e.g. per thread? This presumably is a scalability blocker? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 06:14:48 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 04:14:48 -0000 Subject: [M3devel] joinMu global? In-Reply-To: References: Message-ID: I guess it is probably ok in the scheme of things, since creating threads is also fairly serialized.:( - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: joinMu global? Date: Sun, 31 Jul 2016 03:54:51 +0000 Why is joinMu global, instead of e.g. per thread? This presumably is a scalability blocker? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 06:24:18 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 04:24:18 -0000 Subject: [M3devel] A win32 thread test case In-Reply-To: References: <579A646D.3020404@lcwb.coop>, , , Message-ID: Something like attached?Still testing..might be crashing.A lot of the code and patterns come from ThreadPThread.m3. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 02:52:02 +0000 While I agree this is confusing, it is holding together for me and I have mostly coded it up. Repeating myself: In this scheme there are two types of condition variable. There are Modula-3 condition variables and posix-like/schmidt-like condition variables. The posix-like/smidt-like condition variables: Can be implemented directly via posix condition variables (if Windows pre-Vista had them). They are implemented as a slight modification of the Smidt counter method. The differences are that condition.lock is removed. Caller of cond_signal/cond_broadcast is responsible to hold the same lock as he holds when calling cond_wait. Some of the lock/unlock cycles are removed, and I hope therefore all the bugs are fixed. I still have to work through your examples. Modula-3 condition variables are then built on top of pthread-like/schmidt condition variables just like ThreadPThread.m3. This includes "alertable". The separate event previously in the Win32 implementation becomes a boolean plus cond_signal.XPause changes to call XWait, essentially. I was thinking we could reuse the Win32 thread alerting feature, but this alerts for other reasons so maybe isn't approprirate. If this works, we should be able to share more code between the pthread and win32 implementations.We just need a name for this slightly constrained pthread interface, i.e. where the condition variables assume the lock is held by the caller of signal/broadcast (we don't use broadcast). - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 00:47:24 +0000 ThreadPThread.m3 had the same narrow startup race condition. I fixed it.Again I'm not seeing commit mails. On the larger matter: I think there is a solution, in that, if you look atThreadPThread.m3 and how it uses Posix condition variables,in that it does a fair amount of the work itself,you can term this "condition within condition". So I think maybe we can use the counting solutionand 1) merge the locks and/or 2) take the lock in Signal/Broadcast. But I have to look at it more. Specifically, in ThreadPThread.m3: Broadcast: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; next := t.nextWaiter; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Signal: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Becaise they signal while locked, we can implement this limitedform of condition variable, and then use the Modula-3 pthread logicusing that. Make some sense? Essentially we will have almost-posix condition variables,and exactly-posix mutexes, and then we can use just the pthread codeon top of that. Almost-posix condition variable is a posix condition variable, exceptit is guaranteed you will call pthread_cond_signal with the samelock held as when you call pthread_cond_wait. On a pthread system, the almost-posix condition variables and exactly-posixmutexes are just pass through to the underlying pthreads. On a win32 system, an almost-posix condition variable is similarto what we have now, except, because we know this lock is held,"XWait" can remove some of the leave/enter cycles and remove race conditions. Essentially we only need a "monitor", which is one lock and one condition variable. OR, we can use "directed notification" where each thread has an event, and threads are linked through a "wait block" i.e. the condition. This should work well too. It depends on us owing CreateThread, which we already do. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sat, 30 Jul 2016 21:18:08 +0000 I fixed the new_slots problem. I agree there are problems here. I think Java got away with a similar implementation because their interface is a bit different. i.e. the lock and condition are merged, and/or notification requires the caller to take the lock. We can't do this. We allow signal/broadcast w/o holding a lock. We'd have to resort to a global lock I guess. I suspect nobody else uses this solution. I believe we can solve it better and worse than others. Most libraries do not have their own "CreateThread" function.That is, most libraries let you call pthread_create or Win32 CreateThread,and the library will interoperate with any thread that comes its way.And most libraries don't seem to use thread locals in their solution.By "most libraries", I mean the pthreads on win32 package and Boost. Modula-3 isn't clearly this way. This isn't great, but it is the currentsituation. I believe it is fixable.That is, Modula-3 I believe requires using its library to create threads.This is not great for interoperation. You want to be able to be usedby code that isn't away of you, that just creates threads in the "normal" wayfor their platform. And then, either the current state, or a fix I have in mind, can be takenadvantage of to do "directed notify" w/o creating a kernel eventper wait or notify, like other solutions do. As to the fix, to not require threads go through our "CreateThread",I believe on Windows (which is all that is relevant here), we should havea DllMain that allocates thread locals. We can only easily have a DllMain if we are a .dll.Or we can use __declspec(thread).__declspec(thread) works if the .exe is statically linked to the exe or the dll,or on Vista and newer, but not in a .dll loaded by LoadLibrary prior to Vista.Perhaps that is good enough. If we require Vista then we can just drop a bunch of our code and usethe Win32 condition variables presumably.I remain curious how the condition variables work there, i.e. we can'timplement them ourselves, but it is possible they either require kernelsupport or are well beyond our abilities. - Jay > Date: Thu, 28 Jul 2016 15:00:45 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: A win32 thread test case > > I have attached a test case program designed to expose one of the bugs > I think I see in ThreadWin32.m3. It runs correctly on AMD64_LINUX. > I expect it to fail on Windows, but don't have a Windows machine I > can try it on. > > Anybody willing to try it? It's a self-contained Main module. Just > compile and run it. It will announce what it finds. > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ThreadWin32.m3 Type: application/octet-stream Size: 45813 bytes Desc: not available URL: From jay.krell at cornell.edu Sun Jul 31 10:19:55 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 08:19:55 -0000 Subject: [M3devel] Grisu vs. NT386 In-Reply-To: References: , , Message-ID: nope; it compiles but still doesn't workI'll disable it for now, on NT386 (which isn't quite the right condition, since the C backend works there and likely works for Grisu) - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: RE: Grisu vs. NT386 Date: Thu, 28 Jul 2016 07:11:04 +0000 This should all be fixed now in the NT386 backend. Verification welcome. Work could be shifted to the frontend for some of this, perhaps. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: RE: Grisu vs. NT386 Date: Tue, 26 Jul 2016 06:42:28 +0000 I think I can fix the nested helper call easily enough in the backend. But note that NT386 is definitely currently broken since the introduction of Grisu. We might also consider having the frontend always form parameters into temporaries. Not clear if it should do this as needed for a few known cases, like LONGINT multiplication, or for all integer and float operations on all types, despite most of them not requiring function calls. Also not clear wrt longer standing helper functions like div/mod. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: Grisu vs. NT386 Date: Mon, 25 Jul 2016 07:27:46 +0000 Fyi, Grisu reveals at least two problems in the NT386 backend.In particular, it uses LONGINT, and we don't have much code that does. The two problem are that NT386: - generates function calls for e.g. 64bit integer muliplication - doesn't like generating function calls within function calls, i.e. VAR a,b: LONGINT; Foo(a * b); It fails an assert -- when it goes to do the multiply, which is a function call, it asserts that no function call is in progress. For now I'll change Grisu: VAR a,b,t: LONGINT; t := a * b; Foo(t); but fixing NT386 shouldn't be difficult. Really, we should allow for: Foo(Bar(a * b) * b); .. the thing is, the frontend handles this -- it produces temporaries for function return results: t1 := Bar(a * b); Foo(t1 * a * b); but it doesn't do so for things like add/subtract/multiply/divide. Perhaps it should.Decent backends, including possibly NT386, will enregister the temporaries and generate good code anyway. Thoughts? The alternative is NT386 needs to maintain a stack where it doesn't previously -- a stack of pending function calls. 2) Grisu uses loophole between LONGREAL and LONGINT. This is reasonable, but perhaps not previously seen.NT386 requires loopholes that convert between integers and floats to be using REAL, at least for one direction.This seem merely historical and it should allow LONGREAL, or in particular, it should require no size change.Historically NT386 backend maintained an internal stack, of 32bit sized things -- variables and constants.I extended that to be variably sized, i.e. 32bits or 64bits, but this code wasn't updated. It should be trivial, maybe just relaxing the assertion. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 10:25:57 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 08:25:57 -0000 Subject: [M3devel] using interface compiler in m3core/Grisu? Message-ID: Why can't I say this? C:\dev2\cm3.2\m3-libs\m3core\src\float\Common\grisu\Grisu.m3 IMPORT Compiler; PROCEDURE FastDtoa(v : LONGREAL;.. (* temporarily disable for NT386 until problems investigated *) IF Compiler.ThisPlatform = Compiler.Platform.NT386 THEN RETURN FALSE END; new source -> compiling Grisu.m3 Fatal Error: bad version stamps: Grisu.m3 missing export: Compiler.Platform imported by Grisu.m3missing export: Compiler.ThisPlatform imported by Grisu.m3 *** execution of [, ] failed *** ? I know how to workaround in an ugly way -- via m3makefile chosing a different file.. Thank you, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 10:41:00 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 08:41:00 -0000 Subject: [M3devel] using interface compiler in m3core/Grisu? In-Reply-To: References: Message-ID: oh I see, the documentation includes but I removed it years ago as it is kind of a maintenance burden. Kinda. Author: Jay Michael Krell 2010-12-30 05:35:01Committer: Jay Michael Krell 2010-12-30 05:35:01Parent: 1e492adb4f0b0676b3d62aec23da45d197907302 (add OpenBSD/alpha here too)Child: 655eeacc2d2cbe92676cb95840169c81ede4d0e7 (Merge Unicode branch into head. For a summary of the changes, see)Branches: master, remotes/origin/devel_unicode, remotes/origin/masterFollows: Precedes: verification-release, verification-release-x86_64 remove Compiler.ThisPlatform yet another piece of porting work gone There are no uses of this, and any uses would be somewhat dubious, and easy to replace with a compile time check of TARGET in Quake code. See this code for how: - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: using interface compiler in m3core/Grisu? Date: Sun, 31 Jul 2016 08:25:54 +0000 Why can't I say this? C:\dev2\cm3.2\m3-libs\m3core\src\float\Common\grisu\Grisu.m3 IMPORT Compiler; PROCEDURE FastDtoa(v : LONGREAL;.. (* temporarily disable for NT386 until problems investigated *) IF Compiler.ThisPlatform = Compiler.Platform.NT386 THEN RETURN FALSE END; new source -> compiling Grisu.m3 Fatal Error: bad version stamps: Grisu.m3 missing export: Compiler.Platform imported by Grisu.m3missing export: Compiler.ThisPlatform imported by Grisu.m3 *** execution of [, ] failed *** ? I know how to workaround in an ugly way -- via m3makefile chosing a different file.. Thank you, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 11:07:12 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 09:07:12 -0000 Subject: [M3devel] idioms for tracking initialization state and raising errors? Message-ID: Is this a correct idiom?If so, I think it is fairly reasonable. That is: be very willing to "early return" put all cleanup in finally raise errors in finally In particular, I don't want to repeat the cleanup.I don't want to raise before leaving critical sections. I don't intend this to raise within a raise, but onlyfrom the "normal" exit of the finally block. I also don't want extra local booleans, i.e. raise: boolean to indicate failure.Just the one to indicate success. PROCEDURE InitMutex (mutex: Mutex) = VAR lock: PCRITICAL_SECTION := NIL; locked: PCRITICAL_SECTION := NIL; BEGIN IF mutex.initialized THEN RETURN END; TRY lock := NewCriticalSection(); IF lock = NIL THEN RETURN; END; EnterCriticalSection(ADR(initLock)); locked := ADR(initLock); IF mutex.initialized THEN RETURN END; (* We won the race. *) RTHeapRep.RegisterFinalCleanup (mutex, CleanMutex); mutex.lock := lock; lock := NIL; mutex.initialized := TRUE; FINALLY IF locked # NIL THEN LeaveCriticalSection(locked); END; DelCriticalSection(lock); IF NOT mutex.initialized THEN (* Raise after leaving critical section. *) RuntimeError.Raise (RuntimeError.T.OutOfMemory); END; END; END InitMutex; Thank you, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 12:37:34 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 10:37:34 -0000 Subject: [M3devel] A win32 thread test case In-Reply-To: References: <579A646D.3020404@lcwb.coop>, , , , Message-ID: a fallacy here is that..broadcast isn't used, only signal, so the implementation is more general than needed.An even per thread should work. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 04:24:14 +0000 Something like attached?Still testing..might be crashing.A lot of the code and patterns come from ThreadPThread.m3. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 02:52:02 +0000 While I agree this is confusing, it is holding together for me and I have mostly coded it up. Repeating myself: In this scheme there are two types of condition variable. There are Modula-3 condition variables and posix-like/schmidt-like condition variables. The posix-like/smidt-like condition variables: Can be implemented directly via posix condition variables (if Windows pre-Vista had them). They are implemented as a slight modification of the Smidt counter method. The differences are that condition.lock is removed. Caller of cond_signal/cond_broadcast is responsible to hold the same lock as he holds when calling cond_wait. Some of the lock/unlock cycles are removed, and I hope therefore all the bugs are fixed. I still have to work through your examples. Modula-3 condition variables are then built on top of pthread-like/schmidt condition variables just like ThreadPThread.m3. This includes "alertable". The separate event previously in the Win32 implementation becomes a boolean plus cond_signal.XPause changes to call XWait, essentially. I was thinking we could reuse the Win32 thread alerting feature, but this alerts for other reasons so maybe isn't approprirate. If this works, we should be able to share more code between the pthread and win32 implementations.We just need a name for this slightly constrained pthread interface, i.e. where the condition variables assume the lock is held by the caller of signal/broadcast (we don't use broadcast). - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 00:47:24 +0000 ThreadPThread.m3 had the same narrow startup race condition. I fixed it.Again I'm not seeing commit mails. On the larger matter: I think there is a solution, in that, if you look atThreadPThread.m3 and how it uses Posix condition variables,in that it does a fair amount of the work itself,you can term this "condition within condition". So I think maybe we can use the counting solutionand 1) merge the locks and/or 2) take the lock in Signal/Broadcast. But I have to look at it more. Specifically, in ThreadPThread.m3: Broadcast: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; next := t.nextWaiter; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Signal: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Becaise they signal while locked, we can implement this limitedform of condition variable, and then use the Modula-3 pthread logicusing that. Make some sense? Essentially we will have almost-posix condition variables,and exactly-posix mutexes, and then we can use just the pthread codeon top of that. Almost-posix condition variable is a posix condition variable, exceptit is guaranteed you will call pthread_cond_signal with the samelock held as when you call pthread_cond_wait. On a pthread system, the almost-posix condition variables and exactly-posixmutexes are just pass through to the underlying pthreads. On a win32 system, an almost-posix condition variable is similarto what we have now, except, because we know this lock is held,"XWait" can remove some of the leave/enter cycles and remove race conditions. Essentially we only need a "monitor", which is one lock and one condition variable. OR, we can use "directed notification" where each thread has an event, and threads are linked through a "wait block" i.e. the condition. This should work well too. It depends on us owing CreateThread, which we already do. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sat, 30 Jul 2016 21:18:08 +0000 I fixed the new_slots problem. I agree there are problems here. I think Java got away with a similar implementation because their interface is a bit different. i.e. the lock and condition are merged, and/or notification requires the caller to take the lock. We can't do this. We allow signal/broadcast w/o holding a lock. We'd have to resort to a global lock I guess. I suspect nobody else uses this solution. I believe we can solve it better and worse than others. Most libraries do not have their own "CreateThread" function.That is, most libraries let you call pthread_create or Win32 CreateThread,and the library will interoperate with any thread that comes its way.And most libraries don't seem to use thread locals in their solution.By "most libraries", I mean the pthreads on win32 package and Boost. Modula-3 isn't clearly this way. This isn't great, but it is the currentsituation. I believe it is fixable.That is, Modula-3 I believe requires using its library to create threads.This is not great for interoperation. You want to be able to be usedby code that isn't away of you, that just creates threads in the "normal" wayfor their platform. And then, either the current state, or a fix I have in mind, can be takenadvantage of to do "directed notify" w/o creating a kernel eventper wait or notify, like other solutions do. As to the fix, to not require threads go through our "CreateThread",I believe on Windows (which is all that is relevant here), we should havea DllMain that allocates thread locals. We can only easily have a DllMain if we are a .dll.Or we can use __declspec(thread).__declspec(thread) works if the .exe is statically linked to the exe or the dll,or on Vista and newer, but not in a .dll loaded by LoadLibrary prior to Vista.Perhaps that is good enough. If we require Vista then we can just drop a bunch of our code and usethe Win32 condition variables presumably.I remain curious how the condition variables work there, i.e. we can'timplement them ourselves, but it is possible they either require kernelsupport or are well beyond our abilities. - Jay > Date: Thu, 28 Jul 2016 15:00:45 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: A win32 thread test case > > I have attached a test case program designed to expose one of the bugs > I think I see in ThreadWin32.m3. It runs correctly on AMD64_LINUX. > I expect it to fail on Windows, but don't have a Windows machine I > can try it on. > > Anybody willing to try it? It's a self-contained Main module. Just > compile and run it. It will announce what it finds. > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Jul 31 22:45:52 2016 From: jay.krell at cornell.edu (Jay K) Date: Sun, 31 Jul 2016 20:45:52 -0000 Subject: [M3devel] A win32 thread test case In-Reply-To: References: <579A646D.3020404@lcwb.coop>, , , , , Message-ID: I confirmed your test case fails with unmodified ThreadWin32.m3 and passes with my current work in progress. It is attached. I also removed the indirection on critical sections -- exposing the size to Modula3. I went with just an event as I indicated. So the ThreadPThread.m3 code, but with slight changes: - waits specify time to wait until time to wait until - event instead of pthread_cond - same maintenance of list of waiters Reasonable? - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 10:37:30 +0000 a fallacy here is that..broadcast isn't used, only signal, so the implementation is more general than needed.An even per thread should work. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 04:24:14 +0000 Something like attached?Still testing..might be crashing.A lot of the code and patterns come from ThreadPThread.m3. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 02:52:02 +0000 While I agree this is confusing, it is holding together for me and I have mostly coded it up. Repeating myself: In this scheme there are two types of condition variable. There are Modula-3 condition variables and posix-like/schmidt-like condition variables. The posix-like/smidt-like condition variables: Can be implemented directly via posix condition variables (if Windows pre-Vista had them). They are implemented as a slight modification of the Smidt counter method. The differences are that condition.lock is removed. Caller of cond_signal/cond_broadcast is responsible to hold the same lock as he holds when calling cond_wait. Some of the lock/unlock cycles are removed, and I hope therefore all the bugs are fixed. I still have to work through your examples. Modula-3 condition variables are then built on top of pthread-like/schmidt condition variables just like ThreadPThread.m3. This includes "alertable". The separate event previously in the Win32 implementation becomes a boolean plus cond_signal.XPause changes to call XWait, essentially. I was thinking we could reuse the Win32 thread alerting feature, but this alerts for other reasons so maybe isn't approprirate. If this works, we should be able to share more code between the pthread and win32 implementations.We just need a name for this slightly constrained pthread interface, i.e. where the condition variables assume the lock is held by the caller of signal/broadcast (we don't use broadcast). - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sun, 31 Jul 2016 00:47:24 +0000 ThreadPThread.m3 had the same narrow startup race condition. I fixed it.Again I'm not seeing commit mails. On the larger matter: I think there is a solution, in that, if you look atThreadPThread.m3 and how it uses Posix condition variables,in that it does a fair amount of the work itself,you can term this "condition within condition". So I think maybe we can use the counting solutionand 1) merge the locks and/or 2) take the lock in Signal/Broadcast. But I have to look at it more. Specifically, in ThreadPThread.m3: Broadcast: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; next := t.nextWaiter; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Signal: WITH r = pthread_mutex_lock(t.mutex) DO <*ASSERT r=0*> END; t.nextWaiter := NIL; t.waitingOn := NIL; WITH r = pthread_cond_signal(t.cond) DO <*ASSERT r=0*> END; Becaise they signal while locked, we can implement this limitedform of condition variable, and then use the Modula-3 pthread logicusing that. Make some sense? Essentially we will have almost-posix condition variables,and exactly-posix mutexes, and then we can use just the pthread codeon top of that. Almost-posix condition variable is a posix condition variable, exceptit is guaranteed you will call pthread_cond_signal with the samelock held as when you call pthread_cond_wait. On a pthread system, the almost-posix condition variables and exactly-posixmutexes are just pass through to the underlying pthreads. On a win32 system, an almost-posix condition variable is similarto what we have now, except, because we know this lock is held,"XWait" can remove some of the leave/enter cycles and remove race conditions. Essentially we only need a "monitor", which is one lock and one condition variable. OR, we can use "directed notification" where each thread has an event, and threads are linked through a "wait block" i.e. the condition. This should work well too. It depends on us owing CreateThread, which we already do. - Jay From: jay.krell at cornell.edu To: rodney.m.bates at acm.org; m3devel at elegosoft.com Subject: RE: A win32 thread test case Date: Sat, 30 Jul 2016 21:18:08 +0000 I fixed the new_slots problem. I agree there are problems here. I think Java got away with a similar implementation because their interface is a bit different. i.e. the lock and condition are merged, and/or notification requires the caller to take the lock. We can't do this. We allow signal/broadcast w/o holding a lock. We'd have to resort to a global lock I guess. I suspect nobody else uses this solution. I believe we can solve it better and worse than others. Most libraries do not have their own "CreateThread" function.That is, most libraries let you call pthread_create or Win32 CreateThread,and the library will interoperate with any thread that comes its way.And most libraries don't seem to use thread locals in their solution.By "most libraries", I mean the pthreads on win32 package and Boost. Modula-3 isn't clearly this way. This isn't great, but it is the currentsituation. I believe it is fixable.That is, Modula-3 I believe requires using its library to create threads.This is not great for interoperation. You want to be able to be usedby code that isn't away of you, that just creates threads in the "normal" wayfor their platform. And then, either the current state, or a fix I have in mind, can be takenadvantage of to do "directed notify" w/o creating a kernel eventper wait or notify, like other solutions do. As to the fix, to not require threads go through our "CreateThread",I believe on Windows (which is all that is relevant here), we should havea DllMain that allocates thread locals. We can only easily have a DllMain if we are a .dll.Or we can use __declspec(thread).__declspec(thread) works if the .exe is statically linked to the exe or the dll,or on Vista and newer, but not in a .dll loaded by LoadLibrary prior to Vista.Perhaps that is good enough. If we require Vista then we can just drop a bunch of our code and usethe Win32 condition variables presumably.I remain curious how the condition variables work there, i.e. we can'timplement them ourselves, but it is possible they either require kernelsupport or are well beyond our abilities. - Jay > Date: Thu, 28 Jul 2016 15:00:45 -0500 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: A win32 thread test case > > I have attached a test case program designed to expose one of the bugs > I think I see in ThreadWin32.m3. It runs correctly on AMD64_LINUX. > I expect it to fail on Windows, but don't have a Windows machine I > can try it on. > > Anybody willing to try it? It's a self-contained Main module. Just > compile and run it. It will announce what it finds. > > > -- > Rodney Bates > rodney.m.bates at acm.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ThreadWin32.m3 Type: application/octet-stream Size: 42952 bytes Desc: not available URL: