From wagner at elegosoft.com Mon Mar 1 11:21:52 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 01 Mar 2010 11:21:52 +0100 Subject: [M3devel] Tinderbox AMD64_LINUX and errors in libm3 tests Message-ID: <20100301112152.v325as9n4c48coo0@mail.elegosoft.com> Hi, after some script merges from the release branch to the trunk and adaptation of the tinderbox scripts on birch this platform finally seems to build and test OK again. However, I noticed a compilation error in the libm3 tests: --- tests in /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX--- new source -> compiling PathnameTests.i3 new source -> compiling PathnameTests.m3 new source -> compiling Subr.i3 new source -> compiling TextSubrTbl.i3 new source -> compiling TextSubrTbl.m3 new source -> compiling OSTest.m3 "../src/OSTest.m3", line 298: incompatible types (i) 1 error encountered compilation failed => not building program "OSTest" performing pathname-tests... /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX Fatal Error: package build failed cm3 returned 2 Would anybody care to fix that? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Mon Mar 1 11:42:57 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 1 Mar 2010 10:42:57 +0000 Subject: [M3devel] Tinderbox AMD64_LINUX and errors in libm3 tests In-Reply-To: <20100301112152.v325as9n4c48coo0@mail.elegosoft.com> References: <20100301112152.v325as9n4c48coo0@mail.elegosoft.com> Message-ID: I only ever use m3-sys/m3ests. I haven't learned to use any of the other tests. Surely this is longint. I'll look/fix shortly, though we could really use more participation. - Jay > Date: Mon, 1 Mar 2010 11:21:52 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: [M3devel] Tinderbox AMD64_LINUX and errors in libm3 tests > > Hi, > > after some script merges from the release branch to the trunk and > adaptation of the tinderbox scripts on birch this platform finally > seems to build and test OK again. > > However, I noticed a compilation error in the libm3 tests: > > --- tests in > /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX--- > new source -> compiling PathnameTests.i3 > new source -> compiling PathnameTests.m3 > new source -> compiling Subr.i3 > new source -> compiling TextSubrTbl.i3 > new source -> compiling TextSubrTbl.m3 > new source -> compiling OSTest.m3 > "../src/OSTest.m3", line 298: incompatible types (i) > 1 error encountered > compilation failed => not building program "OSTest" > performing pathname-tests... > /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX > Fatal Error: package build failed > > cm3 returned 2 > > Would anybody care to fix that? > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Mon Mar 1 11:59:27 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 01 Mar 2010 11:59:27 +0100 Subject: [M3devel] Tinderbox AMD64_LINUX and errors in libm3 tests In-Reply-To: References: <20100301112152.v325as9n4c48coo0@mail.elegosoft.com> Message-ID: <20100301115927.xn0vc5tnggksgggc@mail.elegosoft.com> Quoting Jay K : > I only ever use m3-sys/m3ests. I haven't learned to use any of the > other tests. These tests are included in the package tests. Each package may have one or more associated test directories. The tests are run after the package has been built in the test-all-pkgs jobs, both in tinderbox http://www.modula3.com/cm3/logs/package-status.html http://www.modula3.com/cm3/logs/cm3-pkg-report-AMD64_LINUX-2010-03-01-03-31-05.html http://www.modula3.com/cm3/logs/cm3-pkg-report-FreeBSD4-2010-03-01-04-53-25.html ... and in Hudson, for example http://hudson.modula3.com:8080/job/cm3-test-all-pkgs-AMD64_LINUX/ http://hudson.modula3.com:8080/job/cm3-test-all-pkgs-PPC_DARWIN/ http://hudson.modula3.com:8080/job/cm3-test-all-pkgs-FreeBSD4/ ... > Surely this is longint. I'll look/fix shortly, though we could > really use more participation. Great. Thanks, Olaf > - Jay > >> Date: Mon, 1 Mar 2010 11:21:52 +0100 >> From: wagner at elegosoft.com >> To: m3devel at elegosoft.com >> Subject: [M3devel] Tinderbox AMD64_LINUX and errors in libm3 tests >> >> Hi, >> >> after some script merges from the release branch to the trunk and >> adaptation of the tinderbox scripts on birch this platform finally >> seems to build and test OK again. >> >> However, I noticed a compilation error in the libm3 tests: >> >> --- tests in >> /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX--- >> new source -> compiling PathnameTests.i3 >> new source -> compiling PathnameTests.m3 >> new source -> compiling Subr.i3 >> new source -> compiling TextSubrTbl.i3 >> new source -> compiling TextSubrTbl.m3 >> new source -> compiling OSTest.m3 >> "../src/OSTest.m3", line 298: incompatible types (i) >> 1 error encountered >> compilation failed => not building program "OSTest" >> performing pathname-tests... >> /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX >> Fatal Error: package build failed >> >> cm3 returned 2 >> >> Would anybody care to fix that? >> >> Olaf >> -- >> Olaf Wagner -- elego Software Solutions GmbH >> Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany >> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 >> http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin >> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 >> > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 2 14:16:03 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 2 Mar 2010 13:16:03 +0000 Subject: [M3devel] FW: optimizing range checks? In-Reply-To: <20100302125931.619312474001@birch.elegosoft.com> References: <20100302125931.619312474001@birch.elegosoft.com> Message-ID: Hey that's pretty clever. It costs a register, but given: if (b >= constantX|| b <= -constantY) a = 0; The C compiler instead does something like: if ((b + constantY - 1) > (constantX + constantY - 1)) a = 0; This is something the front end could do in many cases.? Adding a constant to eliminate one of the compares and branches is a win. If an x86 compiler will give up a register for this, then it is probably a win everywhere. Granted, it probably requires silent overflow. Oh well. - Jay > Date: Tue, 2 Mar 2010 13:59:31 +0000 > To: m3commit at elegosoft.com > From: jkrell at elego.de > Subject: [M3commit] CVS Update: cm3 > > CVSROOT: /usr/cvs > Changes by: jkrell at birch. 10/03/02 13:59:31 > > Modified files: > cm3/m3-libs/m3core/src/Csupport/Common/: hand.c > > Log message: > remove m3_shift64, which for the record was: > uint64 __stdcall m3_shift64(uint64 a, int b) > { > if (b >= 64 || b <= -64) > a = 0; > else if (b > 0) > a <<= b; > else if (b < 0) > a >>= -b; > return a; > } > > _m3_shift64 at 12: > 00000000: 55 push ebp > 00000001: 8B EC mov ebp,esp > 00000003: 8B 4D 10 mov ecx,dword ptr [ebp+10h] > 00000006: 8D 41 3F lea eax,[ecx+3Fh] > 00000009: 83 F8 7E cmp eax,7Eh > 0000000C: 77 1C ja 0000002A > 0000000E: 8B 45 08 mov eax,dword ptr [ebp+8] > 00000011: 8B 55 0C mov edx,dword ptr [ebp+0Ch] > 00000014: 85 C9 test ecx,ecx > 00000016: 7E 07 jle 0000001F > 00000018: E8 00 00 00 00 call __allshl > 0000001D: EB 0F jmp 0000002E > 0000001F: 7D 0D jge 0000002E > 00000021: F7 D9 neg ecx > 00000023: E8 00 00 00 00 call __aullshr > 00000028: EB 04 jmp 0000002E > 0000002A: 33 C0 xor eax,eax > 0000002C: 33 D2 xor edx,edx > 0000002E: 5D pop ebp > 0000002F: C2 0C 00 ret 0Ch > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 2 15:04:47 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 2 Mar 2010 14:04:47 +0000 Subject: [M3devel] overshifting? Message-ID: PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); I'm guessing the warning is all you're supposed to get here? C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 --- building in NT386 --- new source -> compiling Main.m3 "..\Main.m3", line 894: warning: value out of range *** *** runtime error: *** An enumeration or subrange value was out of range. *** file "..\src\TWordN.m3", line 129 *** Stack trace: FP PC Procedure --------- --------- ------------------------------- 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti nWord\Shift.mg 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 ......... ......... ... more frames ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 19:54:54 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 13:54:54 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: Message-ID: <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. On 2 Mar 2010, at 09:04, Jay K wrote: > PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > > > I'm guessing the warning is all you're supposed to get here? > > > C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > --- building in NT386 --- > new source -> compiling Main.m3 > "..\Main.m3", line 894: warning: value out of range > > *** > *** runtime error: > *** An enumeration or subrange value was out of range. > *** file "..\src\TWordN.m3", line 129 > *** > Stack trace: > FP PC Procedure > --------- --------- ------------------------------- > 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > nWord\Shift.mg > 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > ......... ......... ... more frames ... > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 19:55:38 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 13:55:38 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: Message-ID: <06913ED3-F44E-4A00-822C-8DC1F06E6C06@cs.purdue.edu> PS The warning is there to say that there might be a runtime error. On 2 Mar 2010, at 09:04, Jay K wrote: > PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > > > I'm guessing the warning is all you're supposed to get here? > > > C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > --- building in NT386 --- > new source -> compiling Main.m3 > "..\Main.m3", line 894: warning: value out of range > > *** > *** runtime error: > *** An enumeration or subrange value was out of range. > *** file "..\src\TWordN.m3", line 129 > *** > Stack trace: > FP PC Procedure > --------- --------- ------------------------------- > 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > nWord\Shift.mg > 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > ......... ......... ... more frames ... > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 19:58:05 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 13:58:05 -0500 Subject: [M3devel] FW: optimizing range checks? In-Reply-To: References: <20100302125931.619312474001@birch.elegosoft.com> Message-ID: The front-end is supposed to be machine-independent. It has no idea what the underlying machine will do on overflow. On 2 Mar 2010, at 08:16, Jay K wrote: > Hey that's pretty clever. > It costs a register, but given: > > if (b >= constantX|| b <= -constantY) > a = 0; > > The C compiler instead does something like: > if ((b + constantY - 1) > (constantX + constantY - 1)) > a = 0; > > This is something the front end could do in many cases.? > Adding a constant to eliminate one of the compares and branches is a win. > If an x86 compiler will give up a register for this, then it is probably a win everywhere. > > Granted, it probably requires silent overflow. Oh well. > > - Jay > > > Date: Tue, 2 Mar 2010 13:59:31 +0000 > > To: m3commit at elegosoft.com > > From: jkrell at elego.de > > Subject: [M3commit] CVS Update: cm3 > > > > CVSROOT: /usr/cvs > > Changes by: jkrell at birch. 10/03/02 13:59:31 > > > > Modified files: > > cm3/m3-libs/m3core/src/Csupport/Common/: hand.c > > > > Log message: > > remove m3_shift64, which for the record was: > > uint64 __stdcall m3_shift64(uint64 a, int b) > > { > > if (b >= 64 || b <= -64) > > a = 0; > > else if (b > 0) > > a <<= b; > > else if (b < 0) > > a >>= -b; > > return a; > > } > > > > _m3_shift64 at 12: > > 00000000: 55 push ebp > > 00000001: 8B EC mov ebp,esp > > 00000003: 8B 4D 10 mov ecx,dword ptr [ebp+10h] > > 00000006: 8D 41 3F lea eax,[ecx+3Fh] > > 00000009: 83 F8 7E cmp eax,7Eh > > 0000000C: 77 1C ja 0000002A > > 0000000E: 8B 45 08 mov eax,dword ptr [ebp+8] > > 00000011: 8B 55 0C mov edx,dword ptr [ebp+0Ch] > > 00000014: 85 C9 test ecx,ecx > > 00000016: 7E 07 jle 0000001F > > 00000018: E8 00 00 00 00 call __allshl > > 0000001D: EB 0F jmp 0000002E > > 0000001F: 7D 0D jge 0000002E > > 00000021: F7 D9 neg ecx > > 00000023: E8 00 00 00 00 call __aullshr > > 00000028: EB 04 jmp 0000002E > > 0000002A: 33 C0 xor eax,eax > > 0000002C: 33 D2 xor edx,edx > > 0000002E: 5D pop ebp > > 0000002F: C2 0C 00 ret 0Ch > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 20:50:27 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 14:50:27 -0500 Subject: [M3devel] overshifting? In-Reply-To: <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu> References: <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu> <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu> <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu> Message-ID: <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu> Oh, yes, of course your backend should not blow up on this. On 2 Mar 2010, at 14:46, Tony Hosking wrote: > P.S. Here are the signatures of Shift, RightShift, and LeftShift. > > PROCEDURE Shift (x: T; n: INTEGER): T; > For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n > 0 is like multiplying by 2^(n) > PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > = Shift (x, n) > PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > = Shift (x, -n) > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 2 Mar 2010, at 14:44, Tony Hosking wrote: > >> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. >> >> So, in fact the run-time error is correct! 630 is not in the range [0..63]. >> >> On 2 Mar 2010, at 13:54, Tony Hosking wrote: >> >>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >>> >>> On 2 Mar 2010, at 09:04, Jay K wrote: >>> >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >>>> >>>> >>>> I'm guessing the warning is all you're supposed to get here? >>>> >>>> >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >>>> --- building in NT386 --- >>>> new source -> compiling Main.m3 >>>> "..\Main.m3", line 894: warning: value out of range >>>> >>>> *** >>>> *** runtime error: >>>> *** An enumeration or subrange value was out of range. >>>> *** file "..\src\TWordN.m3", line 129 >>>> *** >>>> Stack trace: >>>> FP PC Procedure >>>> --------- --------- ------------------------------- >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >>>> nWord\Shift.mg >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >>>> ......... ......... ... more frames ... >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 20:44:21 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 14:44:21 -0500 Subject: [M3devel] overshifting? In-Reply-To: <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu> References: <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu> Message-ID: <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. So, in fact the run-time error is correct! 630 is not in the range [0..63]. On 2 Mar 2010, at 13:54, Tony Hosking wrote: > I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > > On 2 Mar 2010, at 09:04, Jay K wrote: > >> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >> >> >> I'm guessing the warning is all you're supposed to get here? >> >> >> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >> --- building in NT386 --- >> new source -> compiling Main.m3 >> "..\Main.m3", line 894: warning: value out of range >> >> *** >> *** runtime error: >> *** An enumeration or subrange value was out of range. >> *** file "..\src\TWordN.m3", line 129 >> *** >> Stack trace: >> FP PC Procedure >> --------- --------- ------------------------------- >> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >> nWord\Shift.mg >> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >> ......... ......... ... more frames ... >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 20:46:36 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 14:46:36 -0500 Subject: [M3devel] overshifting? In-Reply-To: <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu> References: <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu> <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu> Message-ID: <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu> P.S. Here are the signatures of Shift, RightShift, and LeftShift. PROCEDURE Shift (x: T; n: INTEGER): T; For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n > 0 is like multiplying by 2^(n) PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; = Shift (x, n) PROCEDURE RightShift (x: T; n: [0..Size-1]): T; = Shift (x, -n) Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 2 Mar 2010, at 14:44, Tony Hosking wrote: > Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > > So, in fact the run-time error is correct! 630 is not in the range [0..63]. > > On 2 Mar 2010, at 13:54, Tony Hosking wrote: > >> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >> >> On 2 Mar 2010, at 09:04, Jay K wrote: >> >>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >>> >>> >>> I'm guessing the warning is all you're supposed to get here? >>> >>> >>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >>> --- building in NT386 --- >>> new source -> compiling Main.m3 >>> "..\Main.m3", line 894: warning: value out of range >>> >>> *** >>> *** runtime error: >>> *** An enumeration or subrange value was out of range. >>> *** file "..\src\TWordN.m3", line 129 >>> *** >>> Stack trace: >>> FP PC Procedure >>> --------- --------- ------------------------------- >>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >>> nWord\Shift.mg >>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >>> ......... ......... ... more frames ... >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 2 22:09:54 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 2 Mar 2010 21:09:54 +0000 Subject: [M3devel] overshifting? In-Reply-To: <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu> References: , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu> Message-ID: I fixed it. It falls back to generating slower code, which is then not hit because it guaranteeably hits a runtime error ahead of it. Perhaps it should just issue a breakpoint there as the code is not reachable. e.g. in C we have: __declspec(noreturn) abort(); void F1() { abort(); /* breakpoint here */ } Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? - Jay From: hosking at cs.purdue.edu Date: Tue, 2 Mar 2010 14:50:27 -0500 To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com; jay.krell at cornell.edu Subject: Re: [M3devel] overshifting? Oh, yes, of course your backend should not blow up on this. On 2 Mar 2010, at 14:46, Tony Hosking wrote: P.S. Here are the signatures of Shift, RightShift, and LeftShift. PROCEDURE Shift (x: T; n: INTEGER): T; For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n > 0 is like multiplying by 2^(n)PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; = Shift (x, n)PROCEDURE RightShift (x: T; n: [0..Size-1]): T; = Shift (x, -n) Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 2 Mar 2010, at 14:44, Tony Hosking wrote: Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. So, in fact the run-time error is correct! 630 is not in the range [0..63]. On 2 Mar 2010, at 13:54, Tony Hosking wrote: I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. On 2 Mar 2010, at 09:04, Jay K wrote: PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); I'm guessing the warning is all you're supposed to get here? C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 --- building in NT386 --- new source -> compiling Main.m3 "..\Main.m3", line 894: warning: value out of range *** *** runtime error: *** An enumeration or subrange value was out of range. *** file "..\src\TWordN.m3", line 129 *** Stack trace: FP PC Procedure --------- --------- ------------------------------- 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti nWord\Shift.mg 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 ......... ......... ... more frames ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 3 00:53:46 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 2 Mar 2010 23:53:46 +0000 Subject: [M3devel] overshifting? In-Reply-To: References: , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, Message-ID: The front end shouldn't bother asking the backend to shift by overlarge constants, eh? Granted, it could be from Shift(a, LAST(INTEGER) + ....); so the frontend can't always know, even if the backend thinks it does. I need to try some things there -- code that produces a large shift via overflowing constants. Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: PROCEDURE Crash() BEGIN EVAL VAL(2, BOOLEAN); END Crash. mostly these are warnings at compile, runtime errors. But some of these things are now barred by m3back, errors, e.g. EVAL LAST(INTEGER) + 1; Maybe I do need that warning callback after all? - Jay ________________________________ > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Tue, 2 Mar 2010 21:09:54 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] overshifting? > > > > > > > > > I fixed it. It falls back to generating slower code, which is then not hit because > > it guaranteeably hits a runtime error ahead of it. > > Perhaps it should just issue a breakpoint there as the code is not reachable. > > e.g. in C we have: > > > > __declspec(noreturn) abort(); > > > > void F1() > > { > abort(); > > /* breakpoint here */ > > } > > > > Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? > > > > - Jay > > > ________________________________ > > From: hosking at cs.purdue.edu > Date: Tue, 2 Mar 2010 14:50:27 -0500 > To: hosking at cs.purdue.edu > CC: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: Re: [M3devel] overshifting? > > > Oh, yes, of course your backend should not blow up on this. > > > > > > > On 2 Mar 2010, at 14:46, Tony Hosking wrote: > > > > P.S. Here are the signatures of Shift, RightShift, and LeftShift. > > > > PROCEDURE Shift (x: T; n: INTEGER): T; > > For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) > > PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > > = Shift (x, n) > > PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > > = Shift (x, -n) > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue University > > 305 N. University Street | West Lafayette | IN 47907 | USA > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > On 2 Mar 2010, at 14:44, Tony Hosking wrote: > > > > > > Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > > > So, in fact the run-time error is correct! 630 is not in the range [0..63]. > > > > On 2 Mar 2010, at 13:54, Tony Hosking wrote: > > > > > I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > > > > On 2 Mar 2010, at 09:04, Jay K wrote: > > > > PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > > > I'm guessing the warning is all you're supposed to get here? > > > C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > --- building in NT386 --- > new source -> compiling Main.m3 > "..\Main.m3", line 894: warning: value out of range > > *** > *** runtime error: > *** An enumeration or subrange value was out of range. > *** file "..\src\TWordN.m3", line 129 > *** > Stack trace: > FP PC Procedure > --------- --------- ------------------------------- > 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > nWord\Shift.mg > 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > ......... ......... ... more frames ... > > > > > From hosking at cs.purdue.edu Wed Mar 3 04:29:47 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 22:29:47 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, Message-ID: The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. On 2 Mar 2010, at 18:53, Jay K wrote: > > The front end shouldn't bother asking the backend to shift by overlarge constants, eh? > Granted, it could be from > Shift(a, LAST(INTEGER) + ....); > > so the frontend can't always know, even if the backend thinks it does. > I need to try some things there -- code that produces a large shift via overflowing constants. > > Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: > > PROCEDURE Crash() > BEGIN > EVAL VAL(2, BOOLEAN); > END Crash. > > > mostly these are warnings at compile, runtime errors. > But some of these things are now barred by m3back, errors, e.g. > > EVAL LAST(INTEGER) + 1; > > Maybe I do need that warning callback after all? > > > - Jay > > > ________________________________ >> From: jay.krell at cornell.edu >> To: hosking at cs.purdue.edu >> Date: Tue, 2 Mar 2010 21:09:54 +0000 >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] overshifting? >> >> >> >> >> >> >> >> >> I fixed it. It falls back to generating slower code, which is then not hit because >> >> it guaranteeably hits a runtime error ahead of it. >> >> Perhaps it should just issue a breakpoint there as the code is not reachable. >> >> e.g. in C we have: >> >> >> >> __declspec(noreturn) abort(); >> >> >> >> void F1() >> >> { >> abort(); >> >> /* breakpoint here */ >> >> } >> >> >> >> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? >> >> >> >> - Jay >> >> >> ________________________________ >> >> From: hosking at cs.purdue.edu >> Date: Tue, 2 Mar 2010 14:50:27 -0500 >> To: hosking at cs.purdue.edu >> CC: m3devel at elegosoft.com; jay.krell at cornell.edu >> Subject: Re: [M3devel] overshifting? >> >> >> Oh, yes, of course your backend should not blow up on this. >> >> >> >> >> >> >> On 2 Mar 2010, at 14:46, Tony Hosking wrote: >> >> >> >> P.S. Here are the signatures of Shift, RightShift, and LeftShift. >> >> >> >> PROCEDURE Shift (x: T; n: INTEGER): T; >> >> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) >> >> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; >> >> = Shift (x, n) >> >> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; >> >> = Shift (x, -n) >> >> >> >> >> >> Antony Hosking | Associate Professor | Computer Science | Purdue University >> >> 305 N. University Street | West Lafayette | IN 47907 | USA >> >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> >> >> On 2 Mar 2010, at 14:44, Tony Hosking wrote: >> >> >> >> >> >> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. >> >> >> So, in fact the run-time error is correct! 630 is not in the range [0..63]. >> >> >> >> On 2 Mar 2010, at 13:54, Tony Hosking wrote: >> >> >> >> >> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >> >> >> >> On 2 Mar 2010, at 09:04, Jay K wrote: >> >> >> >> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >> >> >> I'm guessing the warning is all you're supposed to get here? >> >> >> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >> --- building in NT386 --- >> new source -> compiling Main.m3 >> "..\Main.m3", line 894: warning: value out of range >> >> *** >> *** runtime error: >> *** An enumeration or subrange value was out of range. >> *** file "..\src\TWordN.m3", line 129 >> *** >> Stack trace: >> FP PC Procedure >> --------- --------- ------------------------------- >> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >> nWord\Shift.mg >> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >> ......... ......... ... more frames ... >> >> >> >> >> From jay.krell at cornell.edu Wed Mar 3 05:21:02 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 3 Mar 2010 04:21:02 +0000 Subject: [M3devel] overshifting? In-Reply-To: References: , ,,<9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, ,,<72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, ,,<48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , Message-ID: The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. Doesn't that seem wrong in some way? front end should warn and then backend can be quiet? Why does the backend know more than frontend? Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? - Jay ---------------------------------------- > From: hosking at cs.purdue.edu > Date: Tue, 2 Mar 2010 22:29:47 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] overshifting? > > The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. > > On 2 Mar 2010, at 18:53, Jay K wrote: > >> >> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? >> Granted, it could be from >> Shift(a, LAST(INTEGER) + ....); >> >> so the frontend can't always know, even if the backend thinks it does. >> I need to try some things there -- code that produces a large shift via overflowing constants. >> >> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: >> >> PROCEDURE Crash() >> BEGIN >> EVAL VAL(2, BOOLEAN); >> END Crash. >> >> >> mostly these are warnings at compile, runtime errors. >> But some of these things are now barred by m3back, errors, e.g. >> >> EVAL LAST(INTEGER) + 1; >> >> Maybe I do need that warning callback after all? >> >> >> - Jay >> >> >> ________________________________ >>> From: jay.krell at cornell.edu >>> To: hosking at cs.purdue.edu >>> Date: Tue, 2 Mar 2010 21:09:54 +0000 >>> CC: m3devel at elegosoft.com >>> Subject: Re: [M3devel] overshifting? >>> >>> >>> >>> >>> >>> >>> >>> >>> I fixed it. It falls back to generating slower code, which is then not hit because >>> >>> it guaranteeably hits a runtime error ahead of it. >>> >>> Perhaps it should just issue a breakpoint there as the code is not reachable. >>> >>> e.g. in C we have: >>> >>> >>> >>> __declspec(noreturn) abort(); >>> >>> >>> >>> void F1() >>> >>> { >>> abort(); >>> >>> /* breakpoint here */ >>> >>> } >>> >>> >>> >>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? >>> >>> >>> >>> - Jay >>> >>> >>> ________________________________ >>> >>> From: hosking at cs.purdue.edu >>> Date: Tue, 2 Mar 2010 14:50:27 -0500 >>> To: hosking at cs.purdue.edu >>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu >>> Subject: Re: [M3devel] overshifting? >>> >>> >>> Oh, yes, of course your backend should not blow up on this. >>> >>> >>> >>> >>> >>> >>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: >>> >>> >>> >>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. >>> >>> >>> >>> PROCEDURE Shift (x: T; n: INTEGER): T; >>> >>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) >>> >>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; >>> >>> = Shift (x, n) >>> >>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; >>> >>> = Shift (x, -n) >>> >>> >>> >>> >>> >>> Antony Hosking | Associate Professor | Computer Science | Purdue University >>> >>> 305 N. University Street | West Lafayette | IN 47907 | USA >>> >>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>> >>> >>> >>> >>> >>> >>> >>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: >>> >>> >>> >>> >>> >>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. >>> >>> >>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. >>> >>> >>> >>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: >>> >>> >>> >>> >>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >>> >>> >>> >>> On 2 Mar 2010, at 09:04, Jay K wrote: >>> >>> >>> >>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >>> >>> >>> I'm guessing the warning is all you're supposed to get here? >>> >>> >>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >>> --- building in NT386 --- >>> new source -> compiling Main.m3 >>> "..\Main.m3", line 894: warning: value out of range >>> >>> *** >>> *** runtime error: >>> *** An enumeration or subrange value was out of range. >>> *** file "..\src\TWordN.m3", line 129 >>> *** >>> Stack trace: >>> FP PC Procedure >>> --------- --------- ------------------------------- >>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >>> nWord\Shift.mg >>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >>> ......... ......... ... more frames ... >>> >>> >>> >>> >>> > From jay.krell at cornell.edu Wed Mar 3 05:23:19 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 3 Mar 2010 04:23:19 +0000 Subject: [M3devel] overshifting? In-Reply-To: References: , , , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , , , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , , , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , Message-ID: > You cannot optimise away operations that may cause a run-time error. We don't. We used to. In many cases we absolutely know if there will be overflow or not. If not, optimize. If so, currently, error. But I think maybe it should warn and just call the runtime error functions. Maybe. And I think this is all portable and can be in the frontend. I'm repeating myself now. - Jay ---------------------------------------- > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] overshifting? > Date: Wed, 3 Mar 2010 04:21:02 +0000 > > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > Doesn't that seem wrong in some way? > front end should warn and then backend can be quiet? > Why does the backend know more than frontend? > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > - Jay > > > ---------------------------------------- >> From: hosking at cs.purdue.edu >> Date: Tue, 2 Mar 2010 22:29:47 -0500 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] overshifting? >> >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. >> >> On 2 Mar 2010, at 18:53, Jay K wrote: >> >>> >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? >>> Granted, it could be from >>> Shift(a, LAST(INTEGER) + ....); >>> >>> so the frontend can't always know, even if the backend thinks it does. >>> I need to try some things there -- code that produces a large shift via overflowing constants. >>> >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: >>> >>> PROCEDURE Crash() >>> BEGIN >>> EVAL VAL(2, BOOLEAN); >>> END Crash. >>> >>> >>> mostly these are warnings at compile, runtime errors. >>> But some of these things are now barred by m3back, errors, e.g. >>> >>> EVAL LAST(INTEGER) + 1; >>> >>> Maybe I do need that warning callback after all? >>> >>> >>> - Jay >>> >>> >>> ________________________________ >>>> From: jay.krell at cornell.edu >>>> To: hosking at cs.purdue.edu >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 >>>> CC: m3devel at elegosoft.com >>>> Subject: Re: [M3devel] overshifting? >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> I fixed it. It falls back to generating slower code, which is then not hit because >>>> >>>> it guaranteeably hits a runtime error ahead of it. >>>> >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. >>>> >>>> e.g. in C we have: >>>> >>>> >>>> >>>> __declspec(noreturn) abort(); >>>> >>>> >>>> >>>> void F1() >>>> >>>> { >>>> abort(); >>>> >>>> /* breakpoint here */ >>>> >>>> } >>>> >>>> >>>> >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? >>>> >>>> >>>> >>>> - Jay >>>> >>>> >>>> ________________________________ >>>> >>>> From: hosking at cs.purdue.edu >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 >>>> To: hosking at cs.purdue.edu >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu >>>> Subject: Re: [M3devel] overshifting? >>>> >>>> >>>> Oh, yes, of course your backend should not blow up on this. >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: >>>> >>>> >>>> >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. >>>> >>>> >>>> >>>> PROCEDURE Shift (x: T; n: INTEGER): T; >>>> >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) >>>> >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; >>>> >>>> = Shift (x, n) >>>> >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; >>>> >>>> = Shift (x, -n) >>>> >>>> >>>> >>>> >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University >>>> >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: >>>> >>>> >>>> >>>> >>>> >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. >>>> >>>> >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. >>>> >>>> >>>> >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: >>>> >>>> >>>> >>>> >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >>>> >>>> >>>> >>>> On 2 Mar 2010, at 09:04, Jay K wrote: >>>> >>>> >>>> >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >>>> >>>> >>>> I'm guessing the warning is all you're supposed to get here? >>>> >>>> >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >>>> --- building in NT386 --- >>>> new source -> compiling Main.m3 >>>> "..\Main.m3", line 894: warning: value out of range >>>> >>>> *** >>>> *** runtime error: >>>> *** An enumeration or subrange value was out of range. >>>> *** file "..\src\TWordN.m3", line 129 >>>> *** >>>> Stack trace: >>>> FP PC Procedure >>>> --------- --------- ------------------------------- >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >>>> nWord\Shift.mg >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >>>> ......... ......... ... more frames ... >>>> >>>> >>>> >>>> >>>> >> From hosking at cs.purdue.edu Wed Mar 3 05:44:22 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 23:44:22 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: , , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , Message-ID: <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu> The front-end does fold constants where it can prove no overflow will occur. I don't think I understand any of the rest of what you are saying. On 2 Mar 2010, at 23:21, Jay K wrote: > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > Doesn't that seem wrong in some way? > front end should warn and then backend can be quiet? > Why does the backend know more than frontend? > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > - Jay > > > ---------------------------------------- >> From: hosking at cs.purdue.edu >> Date: Tue, 2 Mar 2010 22:29:47 -0500 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] overshifting? >> >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. >> >> On 2 Mar 2010, at 18:53, Jay K wrote: >> >>> >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? >>> Granted, it could be from >>> Shift(a, LAST(INTEGER) + ....); >>> >>> so the frontend can't always know, even if the backend thinks it does. >>> I need to try some things there -- code that produces a large shift via overflowing constants. >>> >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: >>> >>> PROCEDURE Crash() >>> BEGIN >>> EVAL VAL(2, BOOLEAN); >>> END Crash. >>> >>> >>> mostly these are warnings at compile, runtime errors. >>> But some of these things are now barred by m3back, errors, e.g. >>> >>> EVAL LAST(INTEGER) + 1; >>> >>> Maybe I do need that warning callback after all? >>> >>> >>> - Jay >>> >>> >>> ________________________________ >>>> From: jay.krell at cornell.edu >>>> To: hosking at cs.purdue.edu >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 >>>> CC: m3devel at elegosoft.com >>>> Subject: Re: [M3devel] overshifting? >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> I fixed it. It falls back to generating slower code, which is then not hit because >>>> >>>> it guaranteeably hits a runtime error ahead of it. >>>> >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. >>>> >>>> e.g. in C we have: >>>> >>>> >>>> >>>> __declspec(noreturn) abort(); >>>> >>>> >>>> >>>> void F1() >>>> >>>> { >>>> abort(); >>>> >>>> /* breakpoint here */ >>>> >>>> } >>>> >>>> >>>> >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? >>>> >>>> >>>> >>>> - Jay >>>> >>>> >>>> ________________________________ >>>> >>>> From: hosking at cs.purdue.edu >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 >>>> To: hosking at cs.purdue.edu >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu >>>> Subject: Re: [M3devel] overshifting? >>>> >>>> >>>> Oh, yes, of course your backend should not blow up on this. >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: >>>> >>>> >>>> >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. >>>> >>>> >>>> >>>> PROCEDURE Shift (x: T; n: INTEGER): T; >>>> >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) >>>> >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; >>>> >>>> = Shift (x, n) >>>> >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; >>>> >>>> = Shift (x, -n) >>>> >>>> >>>> >>>> >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University >>>> >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: >>>> >>>> >>>> >>>> >>>> >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. >>>> >>>> >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. >>>> >>>> >>>> >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: >>>> >>>> >>>> >>>> >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >>>> >>>> >>>> >>>> On 2 Mar 2010, at 09:04, Jay K wrote: >>>> >>>> >>>> >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >>>> >>>> >>>> I'm guessing the warning is all you're supposed to get here? >>>> >>>> >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >>>> --- building in NT386 --- >>>> new source -> compiling Main.m3 >>>> "..\Main.m3", line 894: warning: value out of range >>>> >>>> *** >>>> *** runtime error: >>>> *** An enumeration or subrange value was out of range. >>>> *** file "..\src\TWordN.m3", line 129 >>>> *** >>>> Stack trace: >>>> FP PC Procedure >>>> --------- --------- ------------------------------- >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >>>> nWord\Shift.mg >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >>>> ......... ......... ... more frames ... >>>> >>>> >>>> >>>> >>>> >> From jay.krell at cornell.edu Wed Mar 3 07:00:46 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 3 Mar 2010 06:00:46 +0000 Subject: [M3devel] overshifting? In-Reply-To: <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu> References: , , ,,<9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , ,,<72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , ,,<48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , ,,<170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , , , , , , <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu> Message-ID: Why does Word.LeftShift(x , 100) even get to the backend? It should just generate the code to issue a runtime error? (even if x could be zero? I think so.) Why doesn't front warn for -FIRST(INTEGER)? It should, right? But still generate code to compute it. Are these just oversights? - Jay > From: hosking at cs.purdue.edu > Date: Tue, 2 Mar 2010 23:44:22 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] overshifting? > > The front-end does fold constants where it can prove no overflow will occur. I don't think I understand any of the rest of what you are saying. > > On 2 Mar 2010, at 23:21, Jay K wrote: > > > > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > > Doesn't that seem wrong in some way? > > front end should warn and then backend can be quiet? > > Why does the backend know more than frontend? > > > > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > > > > > - Jay > > > > > > ---------------------------------------- > >> From: hosking at cs.purdue.edu > >> Date: Tue, 2 Mar 2010 22:29:47 -0500 > >> To: jay.krell at cornell.edu > >> CC: m3devel at elegosoft.com > >> Subject: Re: [M3devel] overshifting? > >> > >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. > >> > >> On 2 Mar 2010, at 18:53, Jay K wrote: > >> > >>> > >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? > >>> Granted, it could be from > >>> Shift(a, LAST(INTEGER) + ....); > >>> > >>> so the frontend can't always know, even if the backend thinks it does. > >>> I need to try some things there -- code that produces a large shift via overflowing constants. > >>> > >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: > >>> > >>> PROCEDURE Crash() > >>> BEGIN > >>> EVAL VAL(2, BOOLEAN); > >>> END Crash. > >>> > >>> > >>> mostly these are warnings at compile, runtime errors. > >>> But some of these things are now barred by m3back, errors, e.g. > >>> > >>> EVAL LAST(INTEGER) + 1; > >>> > >>> Maybe I do need that warning callback after all? > >>> > >>> > >>> - Jay > >>> > >>> > >>> ________________________________ > >>>> From: jay.krell at cornell.edu > >>>> To: hosking at cs.purdue.edu > >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 > >>>> CC: m3devel at elegosoft.com > >>>> Subject: Re: [M3devel] overshifting? > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> I fixed it. It falls back to generating slower code, which is then not hit because > >>>> > >>>> it guaranteeably hits a runtime error ahead of it. > >>>> > >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. > >>>> > >>>> e.g. in C we have: > >>>> > >>>> > >>>> > >>>> __declspec(noreturn) abort(); > >>>> > >>>> > >>>> > >>>> void F1() > >>>> > >>>> { > >>>> abort(); > >>>> > >>>> /* breakpoint here */ > >>>> > >>>> } > >>>> > >>>> > >>>> > >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? > >>>> > >>>> > >>>> > >>>> - Jay > >>>> > >>>> > >>>> ________________________________ > >>>> > >>>> From: hosking at cs.purdue.edu > >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 > >>>> To: hosking at cs.purdue.edu > >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu > >>>> Subject: Re: [M3devel] overshifting? > >>>> > >>>> > >>>> Oh, yes, of course your backend should not blow up on this. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. > >>>> > >>>> > >>>> > >>>> PROCEDURE Shift (x: T; n: INTEGER): T; > >>>> > >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) > >>>> > >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > >>>> > >>>> = Shift (x, n) > >>>> > >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > >>>> > >>>> = Shift (x, -n) > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University > >>>> > >>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>> > >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > >>>> > >>>> > >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> > >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 09:04, Jay K wrote: > >>>> > >>>> > >>>> > >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > >>>> > >>>> > >>>> I'm guessing the warning is all you're supposed to get here? > >>>> > >>>> > >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > >>>> --- building in NT386 --- > >>>> new source -> compiling Main.m3 > >>>> "..\Main.m3", line 894: warning: value out of range > >>>> > >>>> *** > >>>> *** runtime error: > >>>> *** An enumeration or subrange value was out of range. > >>>> *** file "..\src\TWordN.m3", line 129 > >>>> *** > >>>> Stack trace: > >>>> FP PC Procedure > >>>> --------- --------- ------------------------------- > >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > >>>> nWord\Shift.mg > >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > >>>> ......... ......... ... more frames ... > >>>> > >>>> > >>>> > >>>> > >>>> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 3 16:54:00 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 3 Mar 2010 10:54:00 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: , , , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , , , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , , , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , , , , , , <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu> Message-ID: On 3 Mar 2010, at 01:00, Jay K wrote: > Why does Word.LeftShift(x , 100) even get to the backend? It should just generate the code to issue a runtime error? (even if x could be zero? I think so.) Because the language spec says that a range error (100 does not fit in the range [0..Word.Size-1]) is a run-time error. Note that the code generator's M3CG_Ops don't specify ranges. The range check is in the *interface* for Word. The m3middle and backend M3CG_Ops are intended to be more general. So, M3CG_Ops.shift_left and shift_right are permitted to take an argument that is not range-checked. > Why doesn't front warn for -FIRST(INTEGER)? It should, right? This is independent of the range check above. This is an expression that may or may not evaluate depending on the run-time defaults (checking for overflow or not). The compiler must be conservative and let the run-time influence whether overflow checking is in place or not. Overflow checking is a run-time option. The compiler must defer to the run-time. > But still generate code to compute it. > Are these just oversights? Nope. Intentional. > > - Jay > > > From: hosking at cs.purdue.edu > > Date: Tue, 2 Mar 2010 23:44:22 -0500 > > To: jay.krell at cornell.edu > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] overshifting? > > > > The front-end does fold constants where it can prove no overflow will occur. I don't think I understand any of the rest of what you are saying. > > > > On 2 Mar 2010, at 23:21, Jay K wrote: > > > > > > > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > > > > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > > > > > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > > > > > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > > > Doesn't that seem wrong in some way? > > > front end should warn and then backend can be quiet? > > > Why does the backend know more than frontend? > > > > > > > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > > > > > > > > > - Jay > > > > > > > > > ---------------------------------------- > > >> From: hosking at cs.purdue.edu > > >> Date: Tue, 2 Mar 2010 22:29:47 -0500 > > >> To: jay.krell at cornell.edu > > >> CC: m3devel at elegosoft.com > > >> Subject: Re: [M3devel] overshifting? > > >> > > >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. > > >> > > >> On 2 Mar 2010, at 18:53, Jay K wrote: > > >> > > >>> > > >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? > > >>> Granted, it could be from > > >>> Shift(a, LAST(INTEGER) + ....); > > >>> > > >>> so the frontend can't always know, even if the backend thinks it does. > > >>> I need to try some things there -- code that produces a large shift via overflowing constants. > > >>> > > >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: > > >>> > > >>> PROCEDURE Crash() > > >>> BEGIN > > >>> EVAL VAL(2, BOOLEAN); > > >>> END Crash. > > >>> > > >>> > > >>> mostly these are warnings at compile, runtime errors. > > >>> But some of these things are now barred by m3back, errors, e.g. > > >>> > > >>> EVAL LAST(INTEGER) + 1; > > >>> > > >>> Maybe I do need that warning callback after all? > > >>> > > >>> > > >>> - Jay > > >>> > > >>> > > >>> ________________________________ > > >>>> From: jay.krell at cornell.edu > > >>>> To: hosking at cs.purdue.edu > > >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 > > >>>> CC: m3devel at elegosoft.com > > >>>> Subject: Re: [M3devel] overshifting? > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> I fixed it. It falls back to generating slower code, which is then not hit because > > >>>> > > >>>> it guaranteeably hits a runtime error ahead of it. > > >>>> > > >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. > > >>>> > > >>>> e.g. in C we have: > > >>>> > > >>>> > > >>>> > > >>>> __declspec(noreturn) abort(); > > >>>> > > >>>> > > >>>> > > >>>> void F1() > > >>>> > > >>>> { > > >>>> abort(); > > >>>> > > >>>> /* breakpoint here */ > > >>>> > > >>>> } > > >>>> > > >>>> > > >>>> > > >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? > > >>>> > > >>>> > > >>>> > > >>>> - Jay > > >>>> > > >>>> > > >>>> ________________________________ > > >>>> > > >>>> From: hosking at cs.purdue.edu > > >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 > > >>>> To: hosking at cs.purdue.edu > > >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu > > >>>> Subject: Re: [M3devel] overshifting? > > >>>> > > >>>> > > >>>> Oh, yes, of course your backend should not blow up on this. > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. > > >>>> > > >>>> > > >>>> > > >>>> PROCEDURE Shift (x: T; n: INTEGER): T; > > >>>> > > >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) > > >>>> > > >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > > >>>> > > >>>> = Shift (x, n) > > >>>> > > >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > > >>>> > > >>>> = Shift (x, -n) > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University > > >>>> > > >>>> 305 N. University Street | West Lafayette | IN 47907 | USA > > >>>> > > >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > > >>>> > > >>>> > > >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 09:04, Jay K wrote: > > >>>> > > >>>> > > >>>> > > >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > > >>>> > > >>>> > > >>>> I'm guessing the warning is all you're supposed to get here? > > >>>> > > >>>> > > >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > > >>>> --- building in NT386 --- > > >>>> new source -> compiling Main.m3 > > >>>> "..\Main.m3", line 894: warning: value out of range > > >>>> > > >>>> *** > > >>>> *** runtime error: > > >>>> *** An enumeration or subrange value was out of range. > > >>>> *** file "..\src\TWordN.m3", line 129 > > >>>> *** > > >>>> Stack trace: > > >>>> FP PC Procedure > > >>>> --------- --------- ------------------------------- > > >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > > >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > > >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > > >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > > >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > > >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > > >>>> nWord\Shift.mg > > >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > > >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > > >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > > >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > > >>>> ......... ......... ... more frames ... > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 4 00:52:54 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 3 Mar 2010 23:52:54 +0000 Subject: [M3devel] overshifting? In-Reply-To: References: , , , ,,<9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , , ,,<72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , , ,,<48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , ,,<170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , ,,, ,,, , , , , , , <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu>, , Message-ID: Tony you aren't understanding me. I'm not suggesting to optimize away runtime errors. I understand that part of the spec. It has been referenced multiple times recently. I'm saying that the front end knows often/always without a doubt if an operation will overflow or not. If the operation will not overflow, it can do the work itself. From what you say, it already does. (I'm going to confuse "overflow" and "subrange error" here briefly.) If the operation will overflow, it can just generate code to issue the error. Now, for most operations, it doesn't know how the target responds to overflow, so the second statement isn't quite right. However, if the operation absolutely will overflow, the frontend should warn. It at least misses some cases, e.g. -FIRST(INTEGER). Granted, on a one's complement system, it won't. But it should warn for the sake of all two's complement systems, which is all of them. Let's amend the previous: If the operation absolutely will overflow, the front end should warn, and then just ask CG to generate the usual code, as if the operation wouldn't overflow. I think it misses some warnings currently. I think it's easy to fix and I'll look into it. Basically whenever the TInt operations fail, should warn. Now, let's somewhat replace "overflow" with "subrange error". LeftShift(x, 100) acts the same on all targets. It isn't overflow, it is subrange error. In this case, I assert, the front end should warn, and it shouldn't bother asking the backend to generate code to the shift. -Jay From: hosking at cs.purdue.edu Date: Wed, 3 Mar 2010 10:54:00 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] overshifting? On 3 Mar 2010, at 01:00, Jay K wrote: Why does Word.LeftShift(x , 100) even get to the backend? It should just generate the code to issue a runtime error? (even if x could be zero? I think so.) Because the language spec says that a range error (100 does not fit in the range [0..Word.Size-1]) is a run-time error. Note that the code generator's M3CG_Ops don't specify ranges. The range check is in the *interface* for Word. The m3middle and backend M3CG_Ops are intended to be more general. So, M3CG_Ops.shift_left and shift_right are permitted to take an argument that is not range-checked. Why doesn't front warn for -FIRST(INTEGER)? It should, right? This is independent of the range check above. This is an expression that may or may not evaluate depending on the run-time defaults (checking for overflow or not). The compiler must be conservative and let the run-time influence whether overflow checking is in place or not. Overflow checking is a run-time option. The compiler must defer to the run-time. But still generate code to compute it. Are these just oversights? Nope. Intentional. - Jay > From: hosking at cs.purdue.edu > Date: Tue, 2 Mar 2010 23:44:22 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] overshifting? > > The front-end does fold constants where it can prove no overflow will occur. I don't think I understand any of the rest of what you are saying. > > On 2 Mar 2010, at 23:21, Jay K wrote: > > > > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > > Doesn't that seem wrong in some way? > > front end should warn and then backend can be quiet? > > Why does the backend know more than frontend? > > > > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > > > > > - Jay > > > > > > ---------------------------------------- > >> From: hosking at cs.purdue.edu > >> Date: Tue, 2 Mar 2010 22:29:47 -0500 > >> To: jay.krell at cornell.edu > >> CC: m3devel at elegosoft.com > >> Subject: Re: [M3devel] overshifting? > >> > >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. > >> > >> On 2 Mar 2010, at 18:53, Jay K wrote: > >> > >>> > >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? > >>> Granted, it could be from > >>> Shift(a, LAST(INTEGER) + ....); > >>> > >>> so the frontend can't always know, even if the backend thinks it does. > >>> I need to try some things there -- code that produces a large shift via overflowing constants. > >>> > >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: > >>> > >>> PROCEDURE Crash() > >>> BEGIN > >>> EVAL VAL(2, BOOLEAN); > >>> END Crash. > >>> > >>> > >>> mostly these are warnings at compile, runtime errors. > >>> But some of these things are now barred by m3back, errors, e.g. > >>> > >>> EVAL LAST(INTEGER) + 1; > >>> > >>> Maybe I do need that warning callback after all? > >>> > >>> > >>> - Jay > >>> > >>> > >>> ________________________________ > >>>> From: jay.krell at cornell.edu > >>>> To: hosking at cs.purdue.edu > >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 > >>>> CC: m3devel at elegosoft.com > >>>> Subject: Re: [M3devel] overshifting? > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> I fixed it. It falls back to generating slower code, which is then not hit because > >>>> > >>>> it guaranteeably hits a runtime error ahead of it. > >>>> > >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. > >>>> > >>>> e.g. in C we have: > >>>> > >>>> > >>>> > >>>> __declspec(noreturn) abort(); > >>>> > >>>> > >>>> > >>>> void F1() > >>>> > >>>> { > >>>> abort(); > >>>> > >>>> /* breakpoint here */ > >>>> > >>>> } > >>>> > >>>> > >>>> > >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? > >>>> > >>>> > >>>> > >>>> - Jay > >>>> > >>>> > >>>> ________________________________ > >>>> > >>>> From: hosking at cs.purdue.edu > >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 > >>>> To: hosking at cs.purdue.edu > >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu > >>>> Subject: Re: [M3devel] overshifting? > >>>> > >>>> > >>>> Oh, yes, of course your backend should not blow up on this. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. > >>>> > >>>> > >>>> > >>>> PROCEDURE Shift (x: T; n: INTEGER): T; > >>>> > >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) > >>>> > >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > >>>> > >>>> = Shift (x, n) > >>>> > >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > >>>> > >>>> = Shift (x, -n) > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University > >>>> > >>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>> > >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > >>>> > >>>> > >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> > >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 09:04, Jay K wrote: > >>>> > >>>> > >>>> > >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > >>>> > >>>> > >>>> I'm guessing the warning is all you're supposed to get here? > >>>> > >>>> > >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > >>>> --- building in NT386 --- > >>>> new source -> compiling Main.m3 > >>>> "..\Main.m3", line 894: warning: value out of range > >>>> > >>>> *** > >>>> *** runtime error: > >>>> *** An enumeration or subrange value was out of range. > >>>> *** file "..\src\TWordN.m3", line 129 > >>>> *** > >>>> Stack trace: > >>>> FP PC Procedure > >>>> --------- --------- ------------------------------- > >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > >>>> nWord\Shift.mg > >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > >>>> ......... ......... ... more frames ... > >>>> > >>>> > >>>> > >>>> > >>>> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 4 01:45:07 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 4 Mar 2010 00:45:07 +0000 Subject: [M3devel] set operations parse.c? Message-ID: The flag to setop is maybe yucky, but I think static void m3cg_set_eq (void) { m3cg_set_eq_or_ne(EQ_EXPR); } static void m3cg_set_ne (void) { m3cg_set_eq_or_ne(NE_EXPR); } is still good. There are two functions *very* similar in functionality that vary in only one token. The counterpoint would be, perhaps, if one is proficient enough in maintaining parse.c, that the two functions are small and one could write them up in a flash without referring to examples. One needs that level of proficiency to make more significant changes anyway. (I'll be removing the set_singleton/member functions shortly, assuming reasonable codegen.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 4 04:20:07 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 3 Mar 2010 22:20:07 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: , , , , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , , , , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , , , , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , , , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , , , , , , , , , , , <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu>, , Message-ID: <4671EB9C-6182-462C-B04F-89428155D530@cs.purdue.edu> On 3 Mar 2010, at 18:52, Jay K wrote: > Tony you aren't understanding me. > > > I'm not suggesting to optimize away runtime errors. > I understand that part of the spec. It has been referenced multiple times recently. > > > I'm saying that the front end knows often/always without a doubt if an operation will overflow or not. > If the operation will not overflow, it can do the work itself. > From what you say, it already does. Correct. > (I'm going to confuse "overflow" and "subrange error" here briefly.) But they are distinct concepts! They cannot be merged. > If the operation will overflow, it can just generate code to issue the error. No! Overflow is a run-time error only if enabled at run-time using FloatMode (or if the target checks for overflow). I think you are not understanding me! > Now, for most operations, it doesn't know how the target responds to overflow, so the second statement isn't quite right. > However, if the operation absolutely will overflow, the frontend should warn. It at least misses some cases, e.g. -FIRST(INTEGER). Granted, on a one's complement system, it won't. But it should warn for the sake of all two's complement systems, which is all of them. > > Let's amend the previous: > If the operation absolutely will overflow, the front end should warn, and then just ask CG to generate the usual code, as if the operation wouldn't overflow. I think it misses some warnings currently. I think it's easy to fix and I'll look into it. Basically whenever the TInt operations fail, should warn. > > > Now, let's somewhat replace "overflow" with "subrange error". > LeftShift(x, 100) > acts the same on all targets. It isn't overflow, it is subrange error. > In this case, I assert, the front end should warn, and it shouldn't bother asking the backend to generate code to the shift. > > > -Jay > > > From: hosking at cs.purdue.edu > Date: Wed, 3 Mar 2010 10:54:00 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] overshifting? > > On 3 Mar 2010, at 01:00, Jay K wrote: > > Why does Word.LeftShift(x , 100) even get to the backend? It should just generate the code to issue a runtime error? (even if x could be zero? I think so.) > > Because the language spec says that a range error (100 does not fit in the range [0..Word.Size-1]) is a run-time error. Note that the code generator's M3CG_Ops don't specify ranges. The range check is in the *interface* for Word. The m3middle and backend M3CG_Ops are intended to be more general. So, M3CG_Ops.shift_left and shift_right are permitted to take an argument that is not range-checked. > > Why doesn't front warn for -FIRST(INTEGER)? It should, right? > > This is independent of the range check above. This is an expression that may or may not evaluate depending on the run-time defaults (checking for overflow or not). The compiler must be conservative and let the run-time influence whether overflow checking is in place or not. > > Overflow checking is a run-time option. The compiler must defer to the run-time. > > But still generate code to compute it. > Are these just oversights? > > Nope. Intentional. > > > - Jay > > > From: hosking at cs.purdue.edu > > Date: Tue, 2 Mar 2010 23:44:22 -0500 > > To: jay.krell at cornell.edu > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] overshifting? > > > > The front-end does fold constants where it can prove no overflow will occur. I don't think I understand any of the rest of what you are saying. > > > > On 2 Mar 2010, at 23:21, Jay K wrote: > > > > > > > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > > > > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > > > > > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > > > > > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > > > Doesn't that seem wrong in some way? > > > front end should warn and then backend can be quiet? > > > Why does the backend know more than frontend? > > > > > > > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > > > > > > > > > - Jay > > > > > > > > > ---------------------------------------- > > >> From: hosking at cs.purdue.edu > > >> Date: Tue, 2 Mar 2010 22:29:47 -0500 > > >> To: jay.krell at cornell.edu > > >> CC: m3devel at elegosoft.com > > >> Subject: Re: [M3devel] overshifting? > > >> > > >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. > > >> > > >> On 2 Mar 2010, at 18:53, Jay K wrote: > > >> > > >>> > > >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? > > >>> Granted, it could be from > > >>> Shift(a, LAST(INTEGER) + ....); > > >>> > > >>> so the frontend can't always know, even if the backend thinks it does. > > >>> I need to try some things there -- code that produces a large shift via overflowing constants. > > >>> > > >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: > > >>> > > >>> PROCEDURE Crash() > > >>> BEGIN > > >>> EVAL VAL(2, BOOLEAN); > > >>> END Crash. > > >>> > > >>> > > >>> mostly these are warnings at compile, runtime errors. > > >>> But some of these things are now barred by m3back, errors, e.g. > > >>> > > >>> EVAL LAST(INTEGER) + 1; > > >>> > > >>> Maybe I do need that warning callback after all? > > >>> > > >>> > > >>> - Jay > > >>> > > >>> > > >>> ________________________________ > > >>>> From: jay.krell at cornell.edu > > >>>> To: hosking at cs.purdue.edu > > >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 > > >>>> CC: m3devel at elegosoft.com > > >>>> Subject: Re: [M3devel] overshifting? > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> I fixed it. It falls back to generating slower code, which is then not hit because > > >>>> > > >>>> it guaranteeably hits a runtime error ahead of it. > > >>>> > > >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. > > >>>> > > >>>> e.g. in C we have: > > >>>> > > >>>> > > >>>> > > >>>> __declspec(noreturn) abort(); > > >>>> > > >>>> > > >>>> > > >>>> void F1() > > >>>> > > >>>> { > > >>>> abort(); > > >>>> > > >>>> /* breakpoint here */ > > >>>> > > >>>> } > > >>>> > > >>>> > > >>>> > > >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? > > >>>> > > >>>> > > >>>> > > >>>> - Jay > > >>>> > > >>>> > > >>>> ________________________________ > > >>>> > > >>>> From: hosking at cs.purdue.edu > > >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 > > >>>> To: hosking at cs.purdue.edu > > >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu > > >>>> Subject: Re: [M3devel] overshifting? > > >>>> > > >>>> > > >>>> Oh, yes, of course your backend should not blow up on this. > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. > > >>>> > > >>>> > > >>>> > > >>>> PROCEDURE Shift (x: T; n: INTEGER): T; > > >>>> > > >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) > > >>>> > > >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > > >>>> > > >>>> = Shift (x, n) > > >>>> > > >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > > >>>> > > >>>> = Shift (x, -n) > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University > > >>>> > > >>>> 305 N. University Street | West Lafayette | IN 47907 | USA > > >>>> > > >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > > >>>> > > >>>> > > >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 09:04, Jay K wrote: > > >>>> > > >>>> > > >>>> > > >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > > >>>> > > >>>> > > >>>> I'm guessing the warning is all you're supposed to get here? > > >>>> > > >>>> > > >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > > >>>> --- building in NT386 --- > > >>>> new source -> compiling Main.m3 > > >>>> "..\Main.m3", line 894: warning: value out of range > > >>>> > > >>>> *** > > >>>> *** runtime error: > > >>>> *** An enumeration or subrange value was out of range. > > >>>> *** file "..\src\TWordN.m3", line 129 > > >>>> *** > > >>>> Stack trace: > > >>>> FP PC Procedure > > >>>> --------- --------- ------------------------------- > > >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > > >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > > >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > > >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > > >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > > >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > > >>>> nWord\Shift.mg > > >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > > >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > > >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > > >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > > >>>> ......... ......... ... more frames ... > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 4 04:27:55 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 3 Mar 2010 22:27:55 -0500 Subject: [M3devel] set operations parse.c? In-Reply-To: References: Message-ID: It obscures the simple intent of the eq/ne operations, and conflates them with other similarly simple set operations. Agreed, the implementations now vary in only one token, but I can read them each in a single page of my editor and validate their correctness without having to trace through multiple calls and track a flag value in context along the way. On 3 Mar 2010, at 19:45, Jay K wrote: > The flag to setop is maybe yucky, but I think > static void m3cg_set_eq (void) { m3cg_set_eq_or_ne(EQ_EXPR); } > static void m3cg_set_ne (void) { m3cg_set_eq_or_ne(NE_EXPR); } > > > is still good. > There are two functions *very* similar in functionality that vary in only one token. > > > The counterpoint would be, perhaps, if one is proficient enough > in maintaining parse.c, that the two functions are small and > one could write them up in a flash without referring to examples. EXACTLY! > One needs that level of proficiency to make more significant changes anyway. > (I'll be removing the set_singleton/member functions shortly, assuming > reasonable codegen.) > > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 4 07:44:41 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 4 Mar 2010 06:44:41 +0000 Subject: [M3devel] overshift/overflow Message-ID: Tony, these two questions seemed to go unanswered: 1) What should this code do: Word.LeftShift(..., 100); (* where 100 >= BITSIZE(INTEGER), and similar for Long.LeftShift and shift >= BITSIZE(LONGINT) *) I think m3front should generate a warning and then either generate the same code as today, or something smaller where it only generates code to issue the runtime error. The warning is missing. "Generating the code same as today" is harder on backends, since they have to check for this condition and handle it somehow, though it doesn't matter much what they do, since the runtime error generation will always run and anything after it never will. 2) What should this code do: EVAL -FIRST(INTEGER); I believe the frontend should issue a warning. And then generate the same code as today. Just the warning is missing. 3) Aside, and not a question. I believe, esp. at this point, the notion that overflow checking is a per-thread setting is a mistake. It has "never" been used, save on a small number of targets. It is too late to foist that change on any code thread-wide. The correct thing to do is introduce different interfaces/modules/types/functions which either always do overflow checking, or, perhaps but less likely, new interfaces/modules/types/functions that are runtime configurable, as INTEGER was originally speced. Even the original spec not so great in my opinion. There are too many things to code correctly, let alone worrying about "this" possibly varying underneath you. The overall system is too much a combination of different code to expect just because one set of code wants integer overflow to be an error, that all the code can deal with that. Better to have separate interfaces/functions/types for the different functionality. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 4 07:55:14 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 4 Mar 2010 06:55:14 +0000 Subject: [M3devel] set operations parse.c? In-Reply-To: References: , Message-ID: Any reuse is too obscuring and not worthwhile if the functions already fit on a page? Huh. By that metric, setop, setop2, m3cg_set_compare aren't needed either. Actually I do find setop vs. setop2 a *little* hard to read. And I hope to soon get rid of two out of its three uses pretty soon (set_singleton, member), maybe tonight. The relationship between eq and ne seems among the strongest, except for all the other comparisons. Actually I'm not sure I "like" a lot here. There is or can be enough similarity in the backends, that the frontend probably should just generate either inline code or calls to portable Modula-3 functions, at least lt/le/gt/ge, and to memcmp for eq/ne. singleton/member I'm a bit on the fence. They can be generated portably inline with pretty small code, though that'd take away the ability of m3back to easily know to use bt/bts instructions. You know, my big "fight" is with "too much code". Anything that the frontend can do pretty darn well, is better than doing it in multiple backends. "More higher level operations" are most useful so a "simpler" backend, such as m3back but not gcc, can more easily generate better code. - Jay Subject: Re: set operations parse.c? From: hosking at cs.purdue.edu Date: Wed, 3 Mar 2010 22:27:55 -0500 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu It obscures the simple intent of the eq/ne operations, and conflates them with other similarly simple set operations. Agreed, the implementations now vary in only one token, but I can read them each in a single page of my editor and validate their correctness without having to trace through multiple calls and track a flag value in context along the way. On 3 Mar 2010, at 19:45, Jay K wrote: The flag to setop is maybe yucky, but I think static void m3cg_set_eq (void) { m3cg_set_eq_or_ne(EQ_EXPR); } static void m3cg_set_ne (void) { m3cg_set_eq_or_ne(NE_EXPR); } is still good. There are two functions *very* similar in functionality that vary in only one token. The counterpoint would be, perhaps, if one is proficient enough in maintaining parse.c, that the two functions are small and one could write them up in a flash without referring to examples. EXACTLY! One needs that level of proficiency to make more significant changes anyway. (I'll be removing the set_singleton/member functions shortly, assuming reasonable codegen.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 4 08:43:31 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 4 Mar 2010 02:43:31 -0500 Subject: [M3devel] set operations parse.c? In-Reply-To: References: , Message-ID: On 4 Mar 2010, at 01:55, Jay K wrote: > Any reuse is too obscuring and not worthwhile if the functions already fit on a page? > Huh. Why hide simple code behind a contorted flag-driven procedure abstraction? > By that metric, setop, setop2, m3cg_set_compare aren't needed either. They don't require calling context to understand what they mean. > Actually I do find setop vs. setop2 a *little* hard to read. ;-) > And I hope to soon get rid of two out of its three uses pretty soon (set_singleton, member), maybe tonight. > > > The relationship between eq and ne seems among the strongest, except for all the other comparisons. > > > Actually I'm not sure I "like" a lot here. > There is or can be enough similarity in the backends, that the frontend probably should just generate either inline code or calls to portable Modula-3 functions, at least lt/le/gt/ge, and to memcmp for eq/ne. Too much work for the front-end. It shouldn't have to worry about such things. > singleton/member I'm a bit on the fence. > They can be generated portably inline with pretty small code, though that'd take away the ability of m3back to easily know to use bt/bts instructions. > > > You know, my big "fight" is with "too much code". Anything that the frontend can do pretty darn well, is better than doing it in multiple backends. "More higher level operations" are most useful so a "simpler" backend, such as m3back but not gcc, can more easily generate better code. > > > - Jay > > > Subject: Re: set operations parse.c? > From: hosking at cs.purdue.edu > Date: Wed, 3 Mar 2010 22:27:55 -0500 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > > It obscures the simple intent of the eq/ne operations, and conflates them with other similarly simple set operations. > > Agreed, the implementations now vary in only one token, but I can read them each in a single page of my editor and validate their correctness without having to trace through multiple calls and track a flag value in context along the way. > > On 3 Mar 2010, at 19:45, Jay K wrote: > > The flag to setop is maybe yucky, but I think > static void m3cg_set_eq (void) { m3cg_set_eq_or_ne(EQ_EXPR); } > static void m3cg_set_ne (void) { m3cg_set_eq_or_ne(NE_EXPR); } > > > is still good. > There are two functions *very* similar in functionality that vary in only one token. > > > The counterpoint would be, perhaps, if one is proficient enough > in maintaining parse.c, that the two functions are small and > one could write them up in a flash without referring to examples. > > EXACTLY! > > One needs that level of proficiency to make more significant changes anyway. > (I'll be removing the set_singleton/member functions shortly, assuming > reasonable codegen.) > > > - Jay > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 4 08:57:26 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 4 Mar 2010 02:57:26 -0500 Subject: [M3devel] overshift/overflow In-Reply-To: References: Message-ID: I already answered these questions. On 4 Mar 2010, at 01:44, Jay K wrote: > Tony, these two questions seemed to go unanswered: > > 1) > > What should this code do: > > > Word.LeftShift(..., 100); (* where 100 >= BITSIZE(INTEGER), > and similar for Long.LeftShift and shift >= BITSIZE(LONGINT) *) > > > I think m3front should generate a warning It does generate a warning. > and then either generate the same code as today, > or something smaller where it only generates code > to issue the runtime error. The code for the shift doesn't know that the warning has been generated, since it is all in the checking of the arguments. > The warning is missing. No, it is generated. > "Generating the code same as today" is harder on backends, > since they have to check for this condition and handle it somehow, > though it doesn't matter much what they do, since the runtime > error generation will always run and anything after it never will. Why can't a backend simply generate code for the large shift, as if it had been a call to Word.Shift(..., 100)? > 2) What should this code do: > > > EVAL -FIRST(INTEGER); > > > I believe the frontend should issue a warning. Why? The front-end does not reason about overflow (except when computing compile-time constants). Overflow is a run-time concept!!!!!!!!!!!!! > And then generate the same code as today. > Just the warning is missing. > > > > 3) Aside, and not a question. > I believe, esp. at this point, the notion that overflow checking is a per-thread setting > is a mistake. It has "never" been used, save on a small number of targets. > It is too late to foist that change on any code thread-wide. > The correct thing to do is introduce different interfaces/modules/types/functions > which either always do overflow checking, or, perhaps but less likely, > new interfaces/modules/types/functions that are runtime configurable, as > INTEGER was originally speced. NOOOOO!!!!!!! That will impose an undue expense on targets where such checking is expensive. Targets where trap handlers can be used to catch overflow will enable it to be turned on via FloatMode. Targets that don't support it efficiently don't have to! > Even the original spec not so great in my opinion. > There are too many things to code correctly, let alone worrying about "this" > possibly varying underneath you. The overall system is too much a combination > of different code to expect just because one set of code wants integer overflow > to be an error, that all the code can deal with that. > Better to have separate interfaces/functions/types for the different functionality. > > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 4 10:00:11 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 4 Mar 2010 09:00:11 +0000 Subject: [M3devel] overshift/overflow In-Reply-To: References: , Message-ID: > Word.LeftShift(..., 100); (* where 100 >= BITSIZE(INTEGER), My mistake, it does generate a warning. It also seems to not generate the code to do the shift, good. I had probably hidden 100 behind a function call, inhibiting the front end's checking. Or more likely behind something else. I have to look into something. > Why can't a backend simply generate code for the large shift, as if it had been a call to Word.Shift(..., 100)? It would be dead/unreachable code, but not a big deal. It looks like the front end skips the code when it can figure it out. I'll have to look into this more, but it is acting different/better than I realized. [Jay] EVAL -FIRST(INTEGER); [Jay] I believe the frontend should issue a warning. [Tony] Why? The front-end does not reason about overflow (except when computing compile-time constants). Overflow is a run-time concept!!!!!!!!!!!!! Because it can often easily prove that overflow will occur. It is worth warning about? if I write: a := 1 + 2; does the front end not optimize that to: a := 3; ? Imho it should 1 + 2 provably at compile time does not overflow. > how to enable overflow > The correct thing to do is introduce different interfaces/modules/types/functions > which either always do overflow checking, or, perhaps but less likely, > new interfaces/modules/types/functions that are runtime configurable, as > INTEGER was originally speced. > NOOOOO!!!!!!! That will impose an undue expense on targets where such checking is expensive. I doubt I suggested what you think I did. If someone really needs overflow checking, then it will cost them whatever it costs them, on whatever target they care about. I am NOT suggesting changing any existing interface/module/type. In fact I am proposing that FloatMode's hypothetical feature to enable overflow checking be removed. That "a + b", "a * b" etc., where a is INTEGER, never ever get overflow checking. If anyone needs it, they'd use some as yet to be specified and implemented type/interface/module. Like INTERFACE IntOv; TYPE T = INTEGER; PROCEDURE Add(a, b: T) RAISES Something: T; etc. or maybe a new compiler-builtin type INTEGEROV. Maybe use the word "safe" as it is popular for this sort of thing (search the web for "safeint"). - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From Highjinks at gmx.com Thu Mar 4 22:29:36 2010 From: Highjinks at gmx.com (Chris) Date: Thu, 4 Mar 2010 22:29:36 +0100 (CET) Subject: [M3devel] Referencing opaque C structs in M3 code? Message-ID: <20100304173345.06cb5d69.Highjinks@gmx.com> Alright, this has thrown me for a bit of a loop...no pun intended... Suppose I'm linking my Modula 3 code with a C library where the public API has a declaration like this... typedef struct opaque_foo_t opaque_foo_t; And the type opaque_foo_t is defined in the private part of the library. Do I need to create a seperate representation of the structure for my Modula3 program or can I just create a pointer for it and pass the pointer around? Assuming of course I dont have to actually reference anything inside opaque_foo_t. Would I just create an UNTRACED REF Ctypes.void_star or something similiar? I'm a little puzzled. Any help would be appreciated. -- Chris From mika at async.async.caltech.edu Thu Mar 4 22:49:17 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Thu, 04 Mar 2010 13:49:17 -0800 Subject: [M3devel] Referencing opaque C structs in M3 code? In-Reply-To: <20100304173345.06cb5d69.Highjinks@gmx.com> References: <20100304173345.06cb5d69.Highjinks@gmx.com> Message-ID: <20100304214917.9D0DD1A2080@async.async.caltech.edu> Ctypes.void_star is already what you want. It's supposed to behave like a (void *), which I think is what you want? You can also use ADDRESS but it's a bit less clear, I think. Mika Chris writes: >Alright, this has thrown me for a bit of a loop...no pun intended... > >Suppose I'm linking my Modula 3 code with a C library where the public API has > a declaration like this... > >typedef struct opaque_foo_t opaque_foo_t; > >And the type opaque_foo_t is defined in the private part of the library. > >Do I need to create a seperate representation of the structure for my Modula3 >program or can I just create a pointer for it and pass the pointer around? Ass >uming of course I dont have to actually reference anything inside opaque_foo_t >. Would I just create an UNTRACED REF Ctypes.void_star or something similiar? > >I'm a little puzzled. Any help would be appreciated. > > > >-- >Chris From hosking at cs.purdue.edu Thu Mar 4 22:59:42 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 4 Mar 2010 16:59:42 -0500 Subject: [M3devel] Referencing opaque C structs in M3 code? In-Reply-To: <20100304173345.06cb5d69.Highjinks@gmx.com> References: <20100304173345.06cb5d69.Highjinks@gmx.com> Message-ID: <7CAE16E7-C170-4A0B-B515-2B2DD5493DBE@cs.purdue.edu> You can assume that ADDRESS is entirely interchangeable with a C void *. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 4 Mar 2010, at 16:29, Chris wrote: > Alright, this has thrown me for a bit of a loop...no pun intended... > > Suppose I'm linking my Modula 3 code with a C library where the public API has a declaration like this... > > typedef struct opaque_foo_t opaque_foo_t; > > And the type opaque_foo_t is defined in the private part of the library. > > Do I need to create a seperate representation of the structure for my Modula3 program or can I just create a pointer for it and pass the pointer around? Assuming of course I dont have to actually reference anything inside opaque_foo_t. Would I just create an UNTRACED REF Ctypes.void_star or something similiar? > > I'm a little puzzled. Any help would be appreciated. > > > > -- > Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 5 02:00:37 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 5 Mar 2010 01:00:37 +0000 Subject: [M3devel] optimizing for size or speed? In-Reply-To: References: , , <20100211120309.GB27402@topoi.pooq.com>, Message-ID: I still find these decisions difficult. Esp. when the option is to call a function or not. Esp. for "builtin" stuff like 64bit math and set operations. - Jay ________________________________ > From: jay.krell at cornell.edu > To: hendrik at topoi.pooq.com; m3devel at elegosoft.com > Date: Thu, 11 Feb 2010 12:41:12 +0000 > Subject: Re: [M3devel] optimizing for size or speed? > > > > > > > > > cache -- good point I had forgotten, thanks. > > > > > > Still: > > use the divide instruction, which is smallest, or multiply by reciprocal (which is generally a multiply and a shift) > > (Given a 32x32=>64 multiply operation. x86 doesn't even have 32x32=>32, only 32x32=>64, I believe.) > > Any 32bit division by a constant can be optimized this way and every C compiler knows it. > > > > > > multiply by a constant using multiply instruction, or decompose into some adds? > > The AMD64 optimization guide suggests speed optimizations where they give a sequence for multiplication for every constant up to 32, some are just to use mul. But many are one or two other instructions. > > > > Multiply by 5 is: > > lea eax,[eax+eax*4] > > > > Multiply by 10 is: > > lea eax,[eax+eax*4] > > add eax,eax > > > > > The AMD64 manual even advises to inline 64bit shifts by a non-constant. > > But I can't get Visual C++ to do that. It always calls a function. > > > > > - Jay > > >> Date: Thu, 11 Feb 2010 07:03:09 -0500 >> From: hendrik at topoi.pooq.com >> To: m3devel at elegosoft.com >> Subject: Re: [M3devel] optimizing for size or speed? >> >> On Thu, Feb 11, 2010 at 08:53:08AM +0000, Jay K wrote: >>> >>> There are all kinds of equivalent code sequences. >>> For the maintainer of m3back to chose among. >> >> In case of doubt, go for size; size all by itself costs time in cache >> misses, paging, etc. >> >> Besides, it's possible to measure space. >> >> -- hendrik From jay.krell at cornell.edu Fri Mar 5 14:59:02 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 5 Mar 2010 13:59:02 +0000 Subject: [M3devel] examples In-Reply-To: <20100227235250.mbm07ehhwsokskg4@mail.elegosoft.com> References: , <20100227235250.mbm07ehhwsokskg4@mail.elegosoft.com> Message-ID: Olaf, it looks like I had it backwards. The release branch is as one would want it. You made the change there. Head lags. Should be ported to head but not pressing. Did the files have to move? Oh well. - Jay > Date: Sat, 27 Feb 2010 23:52:50 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] examples > > Quoting Jay K : > > > > > I think the release branch is missing the change to move examples. > > > > Update it? > Yes, please do. > > Olaf > > (I say this based on differences in the scripts.) > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 5 16:06:00 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 5 Mar 2010 15:06:00 +0000 Subject: [M3devel] release status? Message-ID: release status? - cvsupd hang is uninvestigated - NT386 lacks 64bit longint In head, 64bit longint support is quite good, however I found a bug recently, in head, that needs investigation; This code: C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227\Main.m3(129): result64 := Long.Insert(flipA * a64, flipB * b64, m, n); yields: C:\dev2\cm3.2\m3-sys\m3back\src\M3x86.m3(1045): u.Err("Couldn't find var to free in 'free_temp'"); - Also, head vs. release: TInt/TWord/m3front - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 5 16:36:57 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 5 Mar 2010 15:36:57 +0000 Subject: [M3devel] double double double double checking jmp_buf size/alignment In-Reply-To: <20100106162558.7B4101A2079@async.async.caltech.edu> References: , <20100106162558.7B4101A2079@async.async.caltech.edu> Message-ID: Mika, Hm. I would have expected 44 for FreeBSD/x86. Needs slightly more investigation. PPC_DARWIN we agree on. Thanks, - Jay > To: jay.krell at cornell.edu > Date: Wed, 6 Jan 2010 08:25:58 -0800 > From: mika at async.async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] double double double double checking jmp_buf size/alignment > > Forgot the #endif there? > > (81)trs80:~>./a.out > 48 4 > (82)trs80:~>uname -a > FreeBSD trs80.async.caltech.edu 4.11-RELEASE FreeBSD 4.11-RELEASE #3: Mon Nov 21 20:27:08 PST 2005 root at trs80.async.caltech.edu:/usr/src/sys/compile/TRS80 i386 > > [lapdog:~] mika% cc jay.c > [lapdog:~] mika% ./a.out > 768 4 > [lapdog:~] mika% uname -a > Darwin lapdog 8.11.0 Darwin Kernel Version 8.11.0: Wed Oct 10 18:26:00 PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC Power Macintosh powerpc > [lapdog:~] mika% > > Jay K writes: > >--_b0fc625a-8a60-4f2b-9394-0f52cc975894_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >Was truncated!..edited slightly to avoid.. > > > > > >From: jay.krell at cornell.edu > >To: m3devel at elegosoft.com > >Subject: double double double double checking jmp_buf size/alignment > >Date: Wed=2C 6 Jan 2010 12:28:02 +0000 > > > > > > > > > > > > > > > > > >Getting this right is very important. > > Well=2C we can overstate but it is wasteful. > > > > > >So anyone with any time=2C please compile/run this and send the "platform" = > >(uname -a) and the output=2C thanks. > >Or look at m3-libs/m3core/src/C/*/Csetjmp.i3 and > >m3-sys/m3middle/src/Target.m3 and see if all three agree. > > > > > >I have checked a bunch of systems myself but extra checking is good. > > > > > >If you have a system we do not yet or any longer support=2C those are ok to= > >o. > > (e.g. Alpha_*=2C ARM_*=2C MIPS_*=2C *_Irix=2C *_VMS=2C *_Tru64 etc.) > > > >=20 > >#include > >#include > > > >#ifdef __GNUC__ > >#define ALIGN_OF(x) ((int)(sizeof(struct { char a=3B x b=3B }) - sizeof(x))= > >) > >#else > >#define ALIGN_OF(x) ((int)__alignof(x)) > > > >int main() > >{ > > printf("%d %d\n"=2C (int)sizeof(jmp_buf)=2C ALIGN_OF(jmp_buf))=3B > > return 0=3B > >} > > > > > >Thanks=2C > > - Jay > > = > > > >--_b0fc625a-8a60-4f2b-9394-0f52cc975894_ > >Content-Type: text/html; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > > > > > > > > > >Was truncated!..edited slightly to avoid..


>g">From: jay.krell at cornell.edu
To: m3devel at elegosoft.com
Subject: dou= > >ble double double double checking jmp_buf size/alignment
Date: Wed=2C 6 = > >Jan 2010 12:28:02 +0000

> > > > > > > > > > > > > >Getting this right is very important.
 =3B Well=2C we can overstate = > >but it is wasteful.


So anyone with any time=2C please compile/ru= > >n this and send the "platform" (uname -a) and the output=2C thanks.
Or l= > >ook at m3-libs/m3core/src/C/*/Csetjmp.i3 and
m3-sys/m3middle/src/Target.= > >m3 and see if all three agree.


I have checked a bunch of systems= > > myself but extra checking is good.


If you have a system we do n= > >ot yet or any longer support=2C those are ok too.
 =3B(e.g. Alpha_*= > >=2C ARM_*=2C MIPS_*=2C *_Irix=2C *_VMS=2C *_Tru64 etc.)

 =3B
= > >#include <=3Bsetjmp.h>=3B
#include <=3Bstdio.h>=3B

#ifdef= > > __GNUC__
#define ALIGN_OF(x) ((int)(sizeof(struct { char a=3B x b=3B })= > > - sizeof(x)))
#else
#define ALIGN_OF(x) ((int)__alignof(x))

i= > >nt main()
{
 =3B printf("%d %d\n"=2C (int)sizeof(jmp_buf)=2C ALIG= > >N_OF(jmp_buf))=3B
 =3B return 0=3B
}


Thanks=2C
&nbs= > >p=3B- Jay
> >= > > > >--_b0fc625a-8a60-4f2b-9394-0f52cc975894_-- -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 5 16:40:28 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 5 Mar 2010 15:40:28 +0000 Subject: [M3devel] double double double double checking jmp_buf size/alignment In-Reply-To: References: , Message-ID: Randy, thanks, we agree. Target.m3: | Systems.I386_INTERIX => (* Visual C++'s 16, plus two ints, one to say if sigmask saved, and one to possibly save it. *) Jumpbuf_size := 18 * Address.size; | Systems.NT386, Systems.NT386GNU, Systems.I386_NT, Systems.I386_CYGWIN, Systems.I386_MINGW => (* Cygwin is 13, Visual C++ is 16. Interix is 18. Use 18 for interop. Note that Cygwin's setjmp.h header is wrong, off by a factor of 4. Cygwin provides setjmp and _setjmp that resolve the same. Visual C++ provides only _setjmp. Visual C++ also has _setjmp3 that the compiler generates a call to. In fact _setjmp appears to only use 8 ints and _setjmp3 appears to use more. Consider switching to _setjmp3. *) Jumpbuf_size := (18 * Address.size); - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 6 Jan 2010 15:03:02 -0500 Subject: Re: [M3devel] double double double double checking jmp_buf size/alignment Jay: Results on the following platforms are all identical: Windows 2000, Intel Pentium 3: 64 4 Windows XP, Intel Core Duo T2400: 64 4 Windows Vista, Intel Core2 Duo P9600: 64 4 Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, January 06, 2010 8:18 AM To: m3devel Subject: Re: [M3devel] double double double double checking jmp_buf size/alignment Was truncated!..edited slightly to avoid.. From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: double double double double checking jmp_buf size/alignment Date: Wed, 6 Jan 2010 12:28:02 +0000 Getting this right is very important. Well, we can overstate but it is wasteful. So anyone with any time, please compile/run this and send the "platform" (uname -a) and the output, thanks. Or look at m3-libs/m3core/src/C/*/Csetjmp.i3 and m3-sys/m3middle/src/Target.m3 and see if all three agree. I have checked a bunch of systems myself but extra checking is good. If you have a system we do not yet or any longer support, those are ok too. (e.g. Alpha_*, ARM_*, MIPS_*, *_Irix, *_VMS, *_Tru64 etc.) #include #include #ifdef __GNUC__ #define ALIGN_OF(x) ((int)(sizeof(struct { char a; x b; }) - sizeof(x))) #else #define ALIGN_OF(x) ((int)__alignof(x)) int main() { printf("%d %d\n", (int)sizeof(jmp_buf), ALIGN_OF(jmp_buf)); return 0; } Thanks, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Fri Mar 5 21:38:28 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Fri, 5 Mar 2010 15:38:28 -0500 Subject: [M3devel] optimizing for size or speed? In-Reply-To: References: Message-ID: <20100305203828.GA17524@topoi.pooq.com> On Fri, Mar 05, 2010 at 01:00:37AM +0000, Jay K wrote: > > I still find these decisions difficult. > Esp. when the option is to call a function or not. > Esp. for "builtin" stuff like 64bit math and set operations. > > - Jay That's right. They can be difficult. -- hendrik From hendrik at topoi.pooq.com Fri Mar 5 22:03:10 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Fri, 5 Mar 2010 16:03:10 -0500 Subject: [M3devel] Referencing opaque C structs in M3 code? In-Reply-To: <7CAE16E7-C170-4A0B-B515-2B2DD5493DBE@cs.purdue.edu> References: <20100304173345.06cb5d69.Highjinks@gmx.com> <7CAE16E7-C170-4A0B-B515-2B2DD5493DBE@cs.purdue.edu> Message-ID: <20100305210310.GB17524@topoi.pooq.com> On Thu, Mar 04, 2010 at 04:59:42PM -0500, Tony Hosking wrote: > You can assume that ADDRESS is entirely interchangeable with a C void *. What struct opaque_foo_t provides that void doesn't is some further type-checking -- struct opaque_foo_t is a different type from struct other_foo_t. Presumably you should use Modula 3 techniques for making an opaque types in your interface. - hendrik > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 4 Mar 2010, at 16:29, Chris wrote: > > > Alright, this has thrown me for a bit of a loop...no pun intended... > > > > Suppose I'm linking my Modula 3 code with a C library where the public API has a declaration like this... > > > > typedef struct opaque_foo_t opaque_foo_t; > > > > And the type opaque_foo_t is defined in the private part of the library. > > > > Do I need to create a seperate representation of the structure for my Modula3 program or can I just create a pointer for it and pass the pointer around? Assuming of course I dont have to actually reference anything inside opaque_foo_t. Would I just create an UNTRACED REF Ctypes.void_star or something similiar? > > > > I'm a little puzzled. Any help would be appreciated. > > > > > > > > -- > > Chris > From hendrik at topoi.pooq.com Sat Mar 6 07:22:08 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Sat, 6 Mar 2010 01:22:08 -0500 Subject: [M3devel] RC4 Message-ID: <20100306062208.GA16974@topoi.pooq.com> I installed a few packages of RC4 a while ago for Debiaan Squeeze Linux on intel 32-bit, and have had no problems with them. They work just fine, I installed the core package, and then gui and m3gdb. WHile installing I noticed the following: The installation instructions told me to use a new empty directory to unpack packages. Is it a new empty directory for each package, or can I use the same one for all packages? If I use the same one, should I empty it before each package? And should this be different from the directory where I unpacked cm3-bin-ws-m3devtool-TARGET-VERSION.tgz? I made a new directory to unpack each package, but the instructions should be more explicit. There should be some mention which packages depend on which others. I suspect, for example, that juno requires gui. Such constraints should be mentioned in the installatioin instructions. -- hendrik From hendrik at topoi.pooq.com Sat Mar 6 07:28:06 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Sat, 6 Mar 2010 01:28:06 -0500 Subject: [M3devel] FW: optimizing range checks? In-Reply-To: References: <20100302125931.619312474001@birch.elegosoft.com> Message-ID: <20100306062806.GB16974@topoi.pooq.com> On Tue, Mar 02, 2010 at 01:16:03PM +0000, Jay K wrote: > > Hey that's pretty clever. > > It costs a register, but given: > > > > if (b >= constantX|| b <= -constantY) > a = 0; > > > > The C compiler instead does something like: > > if ((b + constantY - 1) > (constantX + constantY - 1)) > > a = 0; > > > > This is something the front end could do in many cases.? > > Adding a constant to eliminate one of the compares and branches is a win. > > If an x86 compiler will give up a register for this, then it is probably a win everywhere. > > > > Granted, it probably requires silent overflow. Oh well. It requires unsigned arithmetic; in particular the final compare has to be unsigned. -- hendrik From hosking at cs.purdue.edu Sat Mar 6 16:12:47 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sat, 6 Mar 2010 10:12:47 -0500 Subject: [M3devel] Referencing opaque C structs in M3 code? In-Reply-To: <20100305210310.GB17524@topoi.pooq.com> References: <20100304173345.06cb5d69.Highjinks@gmx.com> <7CAE16E7-C170-4A0B-B515-2B2DD5493DBE@cs.purdue.edu> <20100305210310.GB17524@topoi.pooq.com> Message-ID: <20C86CDF-49BA-4CA8-918A-D048F110BB7F@cs.purdue.edu> You can have a TYPE foo = BRANDED "foo" ADDRESS to distinguish from other void types. On 5 Mar 2010, at 16:03, hendrik at topoi.pooq.com wrote: > On Thu, Mar 04, 2010 at 04:59:42PM -0500, Tony Hosking wrote: >> You can assume that ADDRESS is entirely interchangeable with a C void *. > > What struct opaque_foo_t provides that void doesn't is some further > type-checking -- struct opaque_foo_t is a different type from struct > other_foo_t. > > Presumably you should use Modula 3 techniques for making an opaque > types in your interface. > > - hendrik > >> >> Antony Hosking | Associate Professor | Computer Science | Purdue University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> On 4 Mar 2010, at 16:29, Chris wrote: >> >>> Alright, this has thrown me for a bit of a loop...no pun intended... >>> >>> Suppose I'm linking my Modula 3 code with a C library where the public API has a declaration like this... >>> >>> typedef struct opaque_foo_t opaque_foo_t; >>> >>> And the type opaque_foo_t is defined in the private part of the library. >>> >>> Do I need to create a seperate representation of the structure for my Modula3 program or can I just create a pointer for it and pass the pointer around? Assuming of course I dont have to actually reference anything inside opaque_foo_t. Would I just create an UNTRACED REF Ctypes.void_star or something similiar? >>> >>> I'm a little puzzled. Any help would be appreciated. >>> >>> >>> >>> -- >>> Chris >> From jay.krell at cornell.edu Sun Mar 7 10:12:38 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 7 Mar 2010 09:12:38 +0000 Subject: [M3devel] word.insert/extract warnings/optimizations Message-ID: IO.PutInt(Word.Insert(a,b,20,20)); IO.PutInt(Word.Extract(1,1,50)); IO.PutInt(Word.Extract(1,30,30)); Tony, there's discrepancy between Word.Extract and Word.Insert. m3front does a better job of checking and optimizing Word.Extract. For Insert it always does a runtime add, and a runtime range check against the number of bits in integer. It isn't wrong, just could be better. For Extract, it checks either constant against the number of bits, if they are both constant, it checks the sum and calls extract_mn. If they are both constant and the sum is too large, it just generates code to cause a range fault at runtime check_hi(1 vs. 0). Probably that could be a little more direct. If one is constant it calls extract_n, or a slightly optimized use of extract if the other is constant. I'll probably tackle this soon. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 7 12:25:10 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 7 Mar 2010 11:25:10 +0000 Subject: [M3devel] -no-m3ship-resolution Message-ID: This is head. Two problems here. One the feature apparently not fully working. I should look at that. Two the test is too platform specific -- lib.lib vs. liblib.a. Maybe reasonable to support some text substitution over the results to address that. Or heck, maybe abstract out the "naming conventions"? Maybe too difficult and too little gain. I'm surprised even have TARGET like that. The main point I think is the install root. --- ../src/p2/p221/stdout.build 2009-12-15 03:04:01.000000000 -0800 +++ ../src/p2/p221/NT386/stdout.build 2010-03-07 03:11:39.531250000 -0800 @@ -1,7 +1,7 @@ -make_dir(PKG_INSTALL & "/p221/" & TARGET) -install_file(".M3EXPORTS", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file("liblib.a", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file("liblib.m3x", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file(".M3WEB", PKG_INSTALL & "/p221/" & TARGET, "0644") -make_dir(PKG_INSTALL & "/p221") -install_file("../Main.m3", PKG_INSTALL & "/p221", "0644") +make_dir("/cm3/pkg/p221/" & TARGET) +install_file(".M3EXPORTS", "/cm3/pkg/p221/" & TARGET, "0644") +install_file("lib.lib", "/cm3/pkg/p221/" & TARGET, "0644") +install_file("lib.m3x", "/cm3/pkg/p221/" & TARGET, "0644") +install_file(".M3WEB", "/cm3/pkg/p221/" & TARGET, "0644") +make_dir("/cm3/pkg/p221") +install_file("../Main.m3", "/cm3/pkg/p221", "0644") --- p222 --- .M3SHIP Library - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 7 12:48:59 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 7 Mar 2010 11:48:59 +0000 Subject: [M3devel] -no-m3ship-resolution In-Reply-To: References: Message-ID: My CM3_INSTALL environment variable has forward slashes. It'd probably work otherwise. One fix is: Index: M3Build.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/cm3/src/M3Build.m3,v retrieving revision 1.31 diff -u -r1.31 M3Build.m3 --- M3Build.m3 4 Sep 2009 10:25:26 -0000 1.31 +++ M3Build.m3 7 Mar 2010 11:40:50 -0000 @@ -133,7 +133,7 @@ (* some more config dependent backward compatibility hacks... *) Quake.Define (t, "M3SEARCH_TABLES", "-T" & M3TFile); Quake.Define (t, "DEFAULT_BUILD_DIR", GetConfig (t, "BUILD_DIR")); - Quake.Define (t, "M3", M3Path.New (GetConfig (t, "BIN_USE"), "cm3")); + Quake.Define (t, "M3", TextUtils.Substitute(M3Path.New (GetConfig (t, "BIN_USE"), "cm3"), "/", M3Path.SlashText)); Quake.Define (t, "PACKAGE_DIR", pkg); t.build_pkg := M3ID.Add (Pathname.Last (pkg)); @@ -143,19 +143,19 @@ (* M3Path.New is used to canonicalize the paths -- to remove dots *) - t.pkg_use := M3Path.New (GetConfig (t, "PKG_USE")); + t.pkg_use := TextUtils.Substitute(M3Path.New (GetConfig (t, "PKG_USE")), "/", M3Path.SlashText); (* not in Quake.Machine - t.bin_use := M3Path.New (GetConfig (t, "BIN_USE")); - t.lib_use := M3Path.New (GetConfig (t, "LIB_USE")); + t.bin_use := TextUtils.Substitute(M3Path.New (GetConfig (t, "BIN_USE")), "/", M3Path.SlashText); + t.lib_use := TextUtils.Substitute(M3Path.New (GetConfig (t, "LIB_USE")), "/", M3Path.SlashText); *) - t.pkg_install := M3Path.New (GetConfig (t, "PKG_INSTALL")); - t.install_root := M3Path.New (GetConfig (t, "INSTALL_ROOT")); - t.bin_install := M3Path.New (GetConfig (t, "BIN_INSTALL")); - t.lib_install := M3Path.New (GetConfig (t, "LIB_INSTALL")); - t.emacs_install := M3Path.New (GetConfig (t, "EMACS_INSTALL")); - t.doc_install := M3Path.New (GetConfig (t, "DOC_INSTALL")); - t.man_install := M3Path.New (GetConfig (t, "MAN_INSTALL")); - t.html_install := M3Path.New (GetConfig (t, "HTML_INSTALL")); + t.pkg_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "PKG_INSTALL")), "/", M3Path.SlashText); + t.install_root := TextUtils.Substitute(M3Path.New (GetConfig (t, "INSTALL_ROOT")), "/", M3Path.SlashText); + t.bin_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "BIN_INSTALL")), "/", M3Path.SlashText); + t.lib_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "LIB_INSTALL")), "/", M3Path.SlashText); + t.emacs_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "EMACS_INSTALL")), "/", M3Path.SlashText); + t.doc_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "DOC_INSTALL")), "/", M3Path.SlashText); + t.man_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "MAN_INSTALL")), "/", M3Path.SlashText); + t.html_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "HTML_INSTALL")), "/", M3Path.SlashText); t.have_pkgtools := GetConfigBool (t, "HAVE_PKGTOOLS"); t.at_SRC := GetConfigBool (t, "AT_SRC"); t.system_liborder := QVal.ToArray (t, ConfigDefn (t, "SYSTEM_LIBORDER").value); but I'm not sure I like that. I think probably we should have parallel variables bin_install_forwardslash, etc., in which \\ is replaced by /. If we compute the paths ourselves on Win32, we use \\. But if user sets them, we leave them alone. Either slash works. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Sun, 7 Mar 2010 11:25:10 +0000 Subject: [M3devel] -no-m3ship-resolution This is head. Two problems here. One the feature apparently not fully working. I should look at that. Two the test is too platform specific -- lib.lib vs. liblib.a. Maybe reasonable to support some text substitution over the results to address that. Or heck, maybe abstract out the "naming conventions"? Maybe too difficult and too little gain. I'm surprised even have TARGET like that. The main point I think is the install root. --- ../src/p2/p221/stdout.build 2009-12-15 03:04:01.000000000 -0800 +++ ../src/p2/p221/NT386/stdout.build 2010-03-07 03:11:39.531250000 -0800 @@ -1,7 +1,7 @@ -make_dir(PKG_INSTALL & "/p221/" & TARGET) -install_file(".M3EXPORTS", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file("liblib.a", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file("liblib.m3x", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file(".M3WEB", PKG_INSTALL & "/p221/" & TARGET, "0644") -make_dir(PKG_INSTALL & "/p221") -install_file("../Main.m3", PKG_INSTALL & "/p221", "0644") +make_dir("/cm3/pkg/p221/" & TARGET) +install_file(".M3EXPORTS", "/cm3/pkg/p221/" & TARGET, "0644") +install_file("lib.lib", "/cm3/pkg/p221/" & TARGET, "0644") +install_file("lib.m3x", "/cm3/pkg/p221/" & TARGET, "0644") +install_file(".M3WEB", "/cm3/pkg/p221/" & TARGET, "0644") +make_dir("/cm3/pkg/p221") +install_file("../Main.m3", "/cm3/pkg/p221", "0644") --- p222 --- .M3SHIP Library - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 7 14:37:28 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 7 Mar 2010 13:37:28 +0000 Subject: [M3devel] test p078 works in release but not head -- NUMBER(open array constant) not constant Message-ID: C:\dev2\cm3.2\m3-sys\m3tests\src\p0\p078 This compiles with release but not head. (* Copyright (C) 1994, Digital Equipment Corporation. *) (* All rights reserved. *) (* See the file COPYRIGHT for a full description. *) MODULE Main; FROM Test IMPORT done, checkI, checkB; CONST Numbers = ARRAY OF INTEGER {2, 3, 5, 7, 11}; First = Numbers [FIRST (Numbers)]; Last = Numbers [LAST (Numbers)]; Number = NUMBER (Numbers); Empty = ARRAY OF INTEGER {}; EFirst = FIRST (Empty); ELast = LAST (Empty); ENumber = NUMBER (Empty); BEGIN checkI (First, 2); checkI (Last, 11); checkI (Number, 5); checkB (EFirst > ELast, TRUE); checkI (ENumber, 0); done (); END Main. new source -> compiling Main.m3 "..\Main.m3", line 14: value is not constant "..\Main.m3", line 19: value is not constant 2 errors encountered compilation failed => not building program "pgm.exe" Fatal Error: package build failed C:\dev2\cm3.2\m3-sys\m3tests\src\p0\p078> It is the lines with "NUMBER". - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 7 15:14:29 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 7 Mar 2010 14:14:29 +0000 Subject: [M3devel] test p078 works in release but not head -- NUMBER(open array constant) not constant In-Reply-To: References: Message-ID: nevermind. Index: Number.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/builtinOps/Number.m3,v retrieving revision 1.5 diff -u -r1.5 Number.m3 --- Number.m3 18 Feb 2010 02:33:09 -0000 1.5 +++ Number.m3 7 Mar 2010 14:11:17 -0000 @@ -102,8 +102,8 @@ IF ArrayExpr.GetBounds (e, min, max) AND TInt.Subtract (max, min, tmp) AND TInt.Add (tmp, TInt.One, num) - AND NOT TInt.LT (num, Target.Integer.max) - AND NOT TInt.LT (Target.Integer.max, num) + AND TInt.GE (num, Target.Integer.min) + AND TInt.LE (num, Target.Integer.max) THEN RETURN IntegerExpr.New (Int.T, num); ELSE RETURN NIL; END; Notice how the full assortment of LE/GE/LT/GT/EQ/NE makes it easier to read and write the code and get it correct. :) Of course, the "double max" stands out as incorrect either way. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu; m3devel at elegosoft.com Date: Sun, 7 Mar 2010 13:37:28 +0000 Subject: [M3devel] test p078 works in release but not head -- NUMBER(open array constant) not constant C:\dev2\cm3.2\m3-sys\m3tests\src\p0\p078 This compiles with release but not head. (* Copyright (C) 1994, Digital Equipment Corporation. *) (* All rights reserved. *) (* See the file COPYRIGHT for a full description. *) MODULE Main; FROM Test IMPORT done, checkI, checkB; CONST Numbers = ARRAY OF INTEGER {2, 3, 5, 7, 11}; First = Numbers [FIRST (Numbers)]; Last = Numbers [LAST (Numbers)]; Number = NUMBER (Numbers); Empty = ARRAY OF INTEGER {}; EFirst = FIRST (Empty); ELast = LAST (Empty); ENumber = NUMBER (Empty); BEGIN checkI (First, 2); checkI (Last, 11); checkI (Number, 5); checkB (EFirst > ELast, TRUE); checkI (ENumber, 0); done (); END Main. new source -> compiling Main.m3 "..\Main.m3", line 14: value is not constant "..\Main.m3", line 19: value is not constant 2 errors encountered compilation failed => not building program "pgm.exe" Fatal Error: package build failed C:\dev2\cm3.2\m3-sys\m3tests\src\p0\p078> It is the lines with "NUMBER". - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Mon Mar 8 10:05:47 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 08 Mar 2010 10:05:47 +0100 Subject: [M3devel] RC4 In-Reply-To: <20100306062208.GA16974@topoi.pooq.com> References: <20100306062208.GA16974@topoi.pooq.com> Message-ID: <20100308100547.2ytic2gw8wcwwo0k@mail.elegosoft.com> Quoting hendrik at topoi.pooq.com: > I installed a few packages of RC4 a while ago for Debiaan Squeeze Linux > on intel 32-bit, and have had no problems with them. They work just > fine, > > I installed the core package, and then gui and m3gdb. Fine. > WHile installing I noticed the following: > > The installation instructions told me to use a new empty > directory to unpack packages. Is it a new empty directory for each > package, or can I use the same one for all packages? If I use the > same one, should I empty it before each package? And should this be > different from the directory where I unpacked > cm3-bin-ws-m3devtool-TARGET-VERSION.tgz? I made a new directory to > unpack each package, but the instructions should be more explicit. I'll try to improve that for RC5. > There should be some mention which packages depend on which others. I > suspect, for example, that juno requires gui. Such constraints should > be mentioned in the installatioin instructions. You mean something like http://www.modula3.com/cm3/releng/collection-juno.html ? Probably the link to the package descriptions is too hidden in the table (last column). Thanks for the feedback, Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Mon Mar 8 10:15:59 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 8 Mar 2010 09:15:59 +0000 Subject: [M3devel] m3cg.i3 reduction? Message-ID: The backend interface has a few aspects that the frontend does not use. Implementations of these are therefore extremely difficult to test and therefore probably don't work. I propose: extract and extract_n should not take a sign_extend:BOOLEAN parameter. It is always false. extract_mn shall retain its sign_extend:BOOLEAN parameter. Though really, it could go too. The front end could just issue divide and the backends could probably figure it out. I like the frontend doing the work though. (Really, if it going to optimize divide by a power of 2, it might be nice to compute reciprocals too...on my list for m3back..) The integer parameters to extract*/insert* should be CARDINAL instead of INTEGER. Or [0..63], with the "63" abstracted out somehow and comments that clarify it is really sometimes only 0..31. The front end already issues range checks for these parameters. The backend shouldn't worry about such wide ranges. Arguably remove insert_m/n and extract_m/n and just have insert/extract. The NT386 backend already notices when parameters on the stack are constants, and the gcc backend probably does too, at least when optimizing. The "specializations" do make it easier for a hypothetical backend simpler than the NT386 to optimize a bit. So I'm on the fence there. zero_n should be removed. It isn't used. Far more radically, I'm suspicious that the use of a stack is a good idea. It'd probably be just as easy or easier, and lead to better code, to have a *sort* of union that encapsulates constants and variables, and pass each "operation" (add/multiply/subtract/insert/extract/etc.) its parameters with the function all. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 8 15:59:41 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 8 Mar 2010 09:59:41 -0500 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: References: Message-ID: <28E061D6-8DB0-4C69-A23D-F460573E16E0@cs.purdue.edu> On 8 Mar 2010, at 04:15, Jay K wrote: > The backend interface has a few aspects that the frontend does not use. > Implementations of these are therefore extremely difficult to test and therefore probably don't work. This is inevitable. Your backend is not implementing an interface that the front-end defines. It is implementing an interface defined in m3middle which is much wider than that used by m3front. For good reason, we should not dumb down m3middle just because m3front doesn't make use of all its operations. Similarly, you should not assume that m3front will not make use of some operations in future. This is good practice for separation of concerns in support of modularity. > I propose: > > > extract and extract_n should not take a sign_extend:BOOLEAN parameter. > It is always false. Only as currently generated by m3front. > extract_mn shall retain its sign_extend:BOOLEAN parameter. For consistency and generality we should retain it for all extract operators. > Though really, it could go too. > The front end could just issue divide and the backends could probably > figure it out. I like the frontend doing the work though. The front-end should not be concerned with optimisation. It's job is semantics, and having extract/insert is important for some targets. > (Really, if it going to optimize divide by a power of 2, it might be nice > to compute reciprocals too...on my list for m3back..) Same. Front-ends should not be worried about optimising. > The integer parameters to extract*/insert* should be CARDINAL instead of INTEGER. > Or [0..63], with the "63" abstracted out somehow and comments that clarify it is > really sometimes only 0..31. That is not a general interface. Just because m3front filters the operands it provides doesn't mean that all upstream clients of m3middle will do so. > The front end already issues range checks for these parameters. > The backend shouldn't worry about such wide ranges. The backend should be prepared for them. > Arguably remove insert_m/n and extract_m/n and just have insert/extract. That is a possibility. But again, some backends might find it convenient to know about constant operands. Let's not constrain things. > The NT386 backend already notices when parameters on the stack are constants, > and the gcc backend probably does too, at least when optimizing. > The "specializations" do make it easier for a hypothetical backend simpler > than the NT386 to optimize a bit. Precisely! > So I'm on the fence there. We should retain them. > Zero_n should be removed. I've considered using it for some initialisation support. We don't want to throw something out that may later be useful. M3CG was designed for generality, and actually even predates the Modula-3 language itself. It is derived from code generators at DEC SRC from the 1980s. > It isn't used. Yet... > Far more radically, I'm suspicious that the use of a stack is a good idea. > It'd probably be just as easy or easier, and lead to better code, to > have a *sort* of union that encapsulates constants and variables, > and pass each "operation" (add/multiply/subtract/insert/extract/etc.) its > parameters with the function all. Stack models versus "registers" have always been argued about in compiling. Witness the Java VM spec's use of a stack as compared to the fact that most Java VMs convert the stack to registers for specific targets. > > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Mar 8 17:17:24 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 8 Mar 2010 16:17:24 +0000 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: <28E061D6-8DB0-4C69-A23D-F460573E16E0@cs.purdue.edu> References: , <28E061D6-8DB0-4C69-A23D-F460573E16E0@cs.purdue.edu> Message-ID: > Front-ends should not be worried about optimising. But it does so much already. It unrolls comparisons of "solid" types to a certain extent. It took me a while to track that down. It converts divides by powers of two into shift/extract. m3front (or m3middle?) could act as "shared code" for multiple backends. Computing the reciprocal is something that would be common to various backends. The computation itself is portable. The backend could chose to use it or not (maybe some processor has a fast divide instruction..or equally slow divide and mutiply...). How about TInt.Reciprocal or somesuch? > doesn't mean that all upstream clients of m3middle will do so What other clients of m3middle could possibly exist? And that really need more functionality than m3front needs? What does insert/extract with negative numbers mean? I'll see if the .i3 files describes it in comments. Please let's not just invent general generalities, that we don't use, and can't test. What m3front needs, we should do. What m3front doesn't need, we should not do. m3middle serves m3front. If/when m3front needs it, do it then. Generalities produce dead untestable buggy code and wasted time. There was definitely some in m3back, e.g. sign extended extract seemed wrong for count = 0. I forgot another suggestion: set_eq and set_ne are implemented the same by the two backends, by calling memcmp. All the other comparisons are the same too, via calling functions. Maybe the frontend should just know about this and call the functions? These are unusually high level operations in the backends, that neither one implements in an at all clever way. - Jay From: hosking at cs.purdue.edu Date: Mon, 8 Mar 2010 09:59:41 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] m3cg.i3 reduction? On 8 Mar 2010, at 04:15, Jay K wrote: The backend interface has a few aspects that the frontend does not use. Implementations of these are therefore extremely difficult to test and therefore probably don't work. This is inevitable. Your backend is not implementing an interface that the front-end defines. It is implementing an interface defined in m3middle which is much wider than that used by m3front. For good reason, we should not dumb down m3middle just because m3front doesn't make use of all its operations. Similarly, you should not assume that m3front will not make use of some operations in future. This is good practice for separation of concerns in support of modularity. I propose: extract and extract_n should not take a sign_extend:BOOLEAN parameter. It is always false. Only as currently generated by m3front. extract_mn shall retain its sign_extend:BOOLEAN parameter. For consistency and generality we should retain it for all extract operators. Though really, it could go too. The front end could just issue divide and the backends could probably figure it out. I like the frontend doing the work though. The front-end should not be concerned with optimisation. It's job is semantics, and having extract/insert is important for some targets. (Really, if it going to optimize divide by a power of 2, it might be nice to compute reciprocals too...on my list for m3back..) Same. Front-ends should not be worried about optimising. The integer parameters to extract*/insert* should be CARDINAL instead of INTEGER. Or [0..63], with the "63" abstracted out somehow and comments that clarify it is really sometimes only 0..31. That is not a general interface. Just because m3front filters the operands it provides doesn't mean that all upstream clients of m3middle will do so. The front end already issues range checks for these parameters. The backend shouldn't worry about such wide ranges. The backend should be prepared for them. Arguably remove insert_m/n and extract_m/n and just have insert/extract. That is a possibility. But again, some backends might find it convenient to know about constant operands. Let's not constrain things. The NT386 backend already notices when parameters on the stack are constants, and the gcc backend probably does too, at least when optimizing. The "specializations" do make it easier for a hypothetical backend simpler than the NT386 to optimize a bit. Precisely! So I'm on the fence there. We should retain them. Zero_n should be removed. I've considered using it for some initialisation support. We don't want to throw something out that may later be useful. M3CG was designed for generality, and actually even predates the Modula-3 language itself. It is derived from code generators at DEC SRC from the 1980s. It isn't used. Yet... Far more radically, I'm suspicious that the use of a stack is a good idea. It'd probably be just as easy or easier, and lead to better code, to have a *sort* of union that encapsulates constants and variables, and pass each "operation" (add/multiply/subtract/insert/extract/etc.) its parameters with the function all. Stack models versus "registers" have always been argued about in compiling. Witness the Java VM spec's use of a stack as compared to the fact that most Java VMs convert the stack to registers for specific targets. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Mon Mar 8 13:25:36 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Mon, 8 Mar 2010 07:25:36 -0500 Subject: [M3devel] RC4 In-Reply-To: <20100308100547.2ytic2gw8wcwwo0k@mail.elegosoft.com> References: <20100306062208.GA16974@topoi.pooq.com> <20100308100547.2ytic2gw8wcwwo0k@mail.elegosoft.com> Message-ID: <20100308122536.GA6668@topoi.pooq.com> On Mon, Mar 08, 2010 at 10:05:47AM +0100, Olaf Wagner wrote: > Quoting hendrik at topoi.pooq.com: > > >I installed a few packages of RC4 a while ago for Debiaan Squeeze Linux > >on intel 32-bit, and have had no problems with them. They work just > >fine, > > > >I installed the core package, and then gui and m3gdb. > > Fine. > > >WHile installing I noticed the following: > > > >The installation instructions told me to use a new empty > >directory to unpack packages. Is it a new empty directory for each > >package, or can I use the same one for all packages? If I use the > >same one, should I empty it before each package? And should this be > >different from the directory where I unpacked > >cm3-bin-ws-m3devtool-TARGET-VERSION.tgz? I made a new directory to > >unpack each package, but the instructions should be more explicit. > > I'll try to improve that for RC5. > > >There should be some mention which packages depend on which others. I > >suspect, for example, that juno requires gui. Such constraints should > >be mentioned in the installatioin instructions. > > You mean something like > > http://www.modula3.com/cm3/releng/collection-juno.html Yes. Something like that! I've never seen that page before. > > ? Probably the link to the package descriptions is too hidden in the table > (last column). What table? > > Thanks for the feedback, > > Olaf It looks like a website navigability problem. As installer, I see the archive descriptions in http://www.modula3.com/cm3/releng/ where the Juno line says, cm3-bin-ws-juno-*.tgz source/binary, application, optional A constraint-based graphical editor and there isn't any mention of a package description. -- hendrik > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > From hendrik at topoi.pooq.com Mon Mar 8 13:37:46 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Mon, 8 Mar 2010 07:37:46 -0500 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: <28E061D6-8DB0-4C69-A23D-F460573E16E0@cs.purdue.edu> References: <28E061D6-8DB0-4C69-A23D-F460573E16E0@cs.purdue.edu> Message-ID: <20100308123746.GB6668@topoi.pooq.com> On Mon, Mar 08, 2010 at 09:59:41AM -0500, Tony Hosking wrote: > > M3CG was designed for generality, and actually even predates the > Modula-3 language itself. It is derived from code generators at DEC > SRC from the 1980s. I've been thinking for a while that the Modula 3 back end and a fair bit of its run-time system are probably good enough to be used by other garbage-collected languages. Almost no one else does a good job of combining parallel processing with garbage collection, for example. -- hendrik From rodney_bates at lcwb.coop Mon Mar 8 18:42:16 2010 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 08 Mar 2010 11:42:16 -0600 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: References: Message-ID: <4B9536F8.7000104@lcwb.coop> It is never realistic to try to thoroughly test a later (than the first) phase of any compiler solely by pumping source code through the earlier phases. Every compiler I have ever worked on had tools to allow hand-coded input to later phases, either already, or else I wrote one, if I had significant responsibility for testing. Sometimes it can be done with a plain old text editor, if the intermediate stream is character-encoded. Similarly, you need to be able dump the intermediate forms of output from earlier phases in human-readable form. Don't we already have at least some of this? Jay K wrote: > The backend interface has a few aspects that the frontend does not use. > Implementations of these are therefore /extremely/ difficult to test and > therefore probably don't work. > > > From hosking at cs.purdue.edu Mon Mar 8 20:32:53 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 8 Mar 2010 14:32:53 -0500 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: <4B9536F8.7000104@lcwb.coop> References: <4B9536F8.7000104@lcwb.coop> Message-ID: On 8 Mar 2010, at 12:42, Rodney M. Bates wrote: > It is never realistic to try to thoroughly test a later (than the first) phase of > any compiler solely by pumping source code through the earlier phases. Every > compiler I have ever worked on had tools to allow hand-coded input to later > phases, either already, or else I wrote one, if I had significant responsibility > for testing. Sometimes it can be done with a plain old text editor, if the > intermediate stream is character-encoded. Indeed. > Similarly, you need to be able dump the intermediate forms of output from earlier > phases in human-readable form. Don't we already have at least some of this? Yes, m3cgcat and friends... > > Jay K wrote: >> The backend interface has a few aspects that the frontend does not use. >> Implementations of these are therefore /extremely/ difficult to test and therefore probably don't work. >> From jay.krell at cornell.edu Tue Mar 9 03:03:27 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 02:03:27 +0000 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: References: , <4B9536F8.7000104@lcwb.coop>, Message-ID: There is no such mode currently for m3back. I think. At least it doesn't operate that way normally. It keeps very little in memory, going fairly directly from function calls to file output. Not entirely, but largely. Not a bad idea though. But maybe I'm wrong here. Maybe the thing to write/read to the gcc backend can also drive m3back? And then I can start with m3front output but edit in variations? That could be handy. Still, these things are give and take. You theorize as to what you might need. Then you write the consumer. Then you discover you might have missed some things and might have put in some unnecessary things. Not all generalizations should be kept. Interfaces should be reviewed for areas that ended up not needed. And you also write the producer and find some things to be easy or difficult. "Build it and they will come", to some extent yes, to some extent no. sign extension was "so easy" that the original authors wrote it, including for non-constants, but I don't think it works for count=0 for example, which the current front end never exercises..not even with constants.. Given my current implementation strategy, which produces less efficient but clear and dependency-free code, it is perhaps easier to extend things back. (still waiting to hear if the larger size bothers anyone, though the tables are gone, that is a reduction; not sure clarity is so important at the assembly level, it is assembly after all) - Jay ---------------------------------------- > From: hosking at cs.purdue.edu > Date: Mon, 8 Mar 2010 14:32:53 -0500 > To: rodney_bates at lcwb.coop > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] m3cg.i3 reduction? > > On 8 Mar 2010, at 12:42, Rodney M. Bates wrote: > >> It is never realistic to try to thoroughly test a later (than the first) phase of >> any compiler solely by pumping source code through the earlier phases. Every >> compiler I have ever worked on had tools to allow hand-coded input to later >> phases, either already, or else I wrote one, if I had significant responsibility >> for testing. Sometimes it can be done with a plain old text editor, if the >> intermediate stream is character-encoded. > > Indeed. > >> Similarly, you need to be able dump the intermediate forms of output from earlier >> phases in human-readable form. Don't we already have at least some of this? > > Yes, m3cgcat and friends... > >> >> Jay K wrote: >>> The backend interface has a few aspects that the frontend does not use. >>> Implementations of these are therefore /extremely/ difficult to test and therefore probably don't work. >>> > From jay.krell at cornell.edu Tue Mar 9 06:09:19 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 05:09:19 +0000 Subject: [M3devel] m3_build vs. build in parse.c? In-Reply-To: <16ADF5CA-B729-40CC-ADAB-5FA170C9084B@cs.purdue.edu> References: , <16ADF5CA-B729-40CC-ADAB-5FA170C9084B@cs.purdue.edu> Message-ID: I understand why use m3_build instead of build. But why ever use build? Just when m3_build has no chance of optimizing? Or an oversight? - Jay ________________________________ > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > CC: m3commit at elegosoft.com > Subject: RE: [M3commit] m3_build vs. build in parse.c? > Date: Thu, 4 Mar 2010 08:46:06 +0000 > > > Why not always call m3_build? > > Why ever call build? > > > > - Jay > >> From: hosking at cs.purdue.edu >> Date: Thu, 4 Mar 2010 02:58:30 -0500 >> To: jay.krell at cornell.edu >> CC: m3commit at elegosoft.com >> Subject: Re: [M3commit] m3_build vs. build in parse.c? >> >> So that we can avoid having to analyse all the constants ourselves. We can simply generate the trees and then have gcc fold them. >> >> On 4 Mar 2010, at 02:16, Jay K wrote: >> >>> Why use one vs. the other? >>> It appears that they are equivalent *except* that m3_build attempts to optimize, >>> but falls back to build if it can't. >>> >>> That is, m3_build calls fold_build. >>> >>> - Jay >>> >>> >>> >>> >> From hosking at cs.purdue.edu Tue Mar 9 06:33:07 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 9 Mar 2010 00:33:07 -0500 Subject: [M3devel] m3_build vs. build in parse.c? In-Reply-To: References: , <16ADF5CA-B729-40CC-ADAB-5FA170C9084B@cs.purdue.edu> Message-ID: <556CE042-9DBD-4432-BBD8-82E0A2DD5801@cs.purdue.edu> Not sure if it break anything or not, but no good reason I know of. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 9 Mar 2010, at 00:09, Jay K wrote: > > I understand why use m3_build instead of build. > But why ever use build? Just when m3_build has > no chance of optimizing? Or an oversight? > > - Jay > > ________________________________ >> From: jay.krell at cornell.edu >> To: hosking at cs.purdue.edu >> CC: m3commit at elegosoft.com >> Subject: RE: [M3commit] m3_build vs. build in parse.c? >> Date: Thu, 4 Mar 2010 08:46:06 +0000 >> >> >> Why not always call m3_build? >> >> Why ever call build? >> >> >> >> - Jay >> >>> From: hosking at cs.purdue.edu >>> Date: Thu, 4 Mar 2010 02:58:30 -0500 >>> To: jay.krell at cornell.edu >>> CC: m3commit at elegosoft.com >>> Subject: Re: [M3commit] m3_build vs. build in parse.c? >>> >>> So that we can avoid having to analyse all the constants ourselves. We can simply generate the trees and then have gcc fold them. >>> >>> On 4 Mar 2010, at 02:16, Jay K wrote: >>> >>>> Why use one vs. the other? >>>> It appears that they are equivalent *except* that m3_build attempts to optimize, >>>> but falls back to build if it can't. >>>> >>>> That is, m3_build calls fold_build. >>>> >>>> - Jay >>>> >>>> >>>> >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 9 15:25:48 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 09 Mar 2010 15:25:48 +0100 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT Message-ID: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com> some windows problems again, probably only help needed... any takers? Olaf ----- Forwarded message from bugs at elego.de ----- Date: Tue, 09 Mar 2010 02:55:44 -0000 From: CM3 Reply-To: CM3 Subject: [CM3] #1082: Windows NT To: @MISSING_DOMAIN #1082: Windows NT -------------------------------------+-------------------------------------- Reporter: ilikesci@? | Owner: wagner Type: sw-test | Status: new Priority: medium | Milestone: Component: misc | Version: 5.8-RC3 Severity: non-critical | Keywords: Relnote: | Org: Estimatedhours: 0 | Hours: 0 Billable: 0 | Totalhours: 0 Internal: 0 | -------------------------------------+-------------------------------------- Htr: Remove msys out of your path. Fix: Env: Microsoft Windows Vista -------------------------------------+-------------------------------------- I am new to modula-3 but have tried a few times to get cm3 working on my MS-Windows machines over the years. This is my new attempt at that. I downloaded the NT386 files and have them unpacked. I have put cm3 in e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is that when I ran cminstall it asked for a tar.exe, gzip.exe, and msys-1.0.dll. I just happened to have msys installed on my computer and dropped them in the folder and cminstall seemed to work. I just wanted to let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and so I ran it per the suggestion on your website. It asked a few questions about what browser I wanted to use and such. The browser does pop up on localhost:3800 but times out. The console spits out: Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 calling start_browser(http://localhost:3800/) starting TCP service start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" http://localhost:3800/ CM3-IDE is shutting down because start_browser() returned TRUE. TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept TCPServer: aborting... I am thinking maybe something else might be running on the port and causing a problem loading page. I copied the startReactor.cmd to the bin directory and tried it and got the following: ------------------------------------------------------------------------------- startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by RCC ------------------------------------------------------------------------------- FATAL ERROR: Unable to find CM3 installation. CM3_ROOT expected in folder C:\cm3 CM3_BIN expected in folder C:\cm3\bin CM3.EXE expected in file C:\cm3\bin\cm3.exe Reactor.EXE expected in file C:\cm3\bin\reactor.exe cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD ------------------------------------------------------------------------------- Which it is installed on e: instead of c: but there is not a reactor.exe file in the bin directory either. From that information I am thinking I should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as e:\cm3. This is FYI and will let you know if I continue and any other experiences that might be of use. Thank You, Micah -- Ticket URL: CM3 Critical Mass Modula3 Compiler ----- End forwarded message ----- -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 9 16:29:58 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 15:29:58 +0000 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT In-Reply-To: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com> References: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com> Message-ID: Try the .msi instead. Try this maybe, I can't find the link on the web page (maybe lost in the crash?) http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/www/installation-windows.html?rev=1.5;content-type=text%2Fplain I should edit/rewrite those. - The bit about "upgrade is pessmistic" isn't true, as my Russian friend says, it is "realistic". There are several breaking changes that motivate doing it the way upgrade does it. - I should steer people to Python instead of cmd (really). - Mentions of 5.2.6 and 5.5.0 and maybe even building from source should probably go away. Leave this for "more beginners". - I need to update the Visual C++ redistributable link (9.0SP1 vs. 9.0). - The config file path is wrong. It is not config-no-install instead of config. - It alludes to too many options, using various versions, building from source or not, etc. But it seems not too bad. It should be retested. The IDE is not what you would normally call an IDE. It is custom web server. Seems like "apples and oranges", but it does make some sense. In some ways it was ahead of its time -- people talk about "application servers" these days, and run C# in the web server to produce html. Just not clear what sorts of apps are suited to this model. There's no editor -- not a bad idea, since everyone prefers what they already have. There's no debugger -- not a bad idea, not like we have the time to write another. You can use Visual Studio or windbg. There is a bit of a project system. However Modula-3 has among the best text file driven command line build systems around, we just need it to handle walking directories, and have it determine the order. It does provide hyperlinking built source, though personally I just use my editor's "find in files" all the time. Maybe someone else can write up something to replace. http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/www/installation-windows.html?rev=1.5;content-type=text%2Fplain#buildm3current Fixing the .cmd to find the .exes should be trivial. Or just have user run the .exes. Have the .exes set any environment variables they need. - Jay > Date: Tue, 9 Mar 2010 15:25:48 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: [M3devel] Fwd: [CM3] #1082: Windows NT > > some windows problems again, probably only help needed... > > any takers? > > Olaf > > ----- Forwarded message from bugs at elego.de ----- > Date: Tue, 09 Mar 2010 02:55:44 -0000 > From: CM3 > Reply-To: CM3 > Subject: [CM3] #1082: Windows NT > To: @MISSING_DOMAIN > > #1082: Windows NT > -------------------------------------+-------------------------------------- > Reporter: ilikesci@? | Owner: wagner > Type: sw-test | Status: new > Priority: medium | Milestone: > Component: misc | Version: 5.8-RC3 > Severity: non-critical | Keywords: > Relnote: | Org: > Estimatedhours: 0 | Hours: 0 > Billable: 0 | Totalhours: 0 > Internal: 0 | > -------------------------------------+-------------------------------------- > Htr: > Remove msys out of your path. > > > Fix: > > > > Env: > Microsoft Windows Vista > > -------------------------------------+-------------------------------------- > I am new to modula-3 but have tried a few times to get cm3 working on my > MS-Windows machines over the years. This is my new attempt at that. I > downloaded the NT386 files and have them unpacked. I have put cm3 in > e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is > that when I ran cminstall it asked for a tar.exe, gzip.exe, and > msys-1.0.dll. I just happened to have msys installed on my computer and > dropped them in the folder and cminstall seemed to work. I just wanted to > let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and > so I ran it per the suggestion on your website. It asked a few questions > about what browser I wanted to use and such. The browser does pop up on > localhost:3800 but times out. The console spits out: > > Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 > calling start_browser(http://localhost:3800/) > starting TCP service > start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" > http://localhost:3800/ > CM3-IDE is shutting down because start_browser() returned TRUE. > TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept > TCPServer: aborting... > > I am thinking maybe something else might be running on the port and > causing a problem loading page. I copied the startReactor.cmd to the bin > directory and tried it and got the following: > > > ------------------------------------------------------------------------------- > startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by > RCC > > ------------------------------------------------------------------------------- > FATAL ERROR: Unable to find CM3 installation. > CM3_ROOT expected in folder C:\cm3 > CM3_BIN expected in folder C:\cm3\bin > CM3.EXE expected in file C:\cm3\bin\cm3.exe > Reactor.EXE expected in file C:\cm3\bin\reactor.exe > cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD > > ------------------------------------------------------------------------------- > Which it is installed on e: instead of c: but there is not a reactor.exe > file in the bin directory either. From that information I am thinking I > should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as > e:\cm3. > > This is FYI and will let you know if I continue and any other experiences > that might be of use. > > Thank You, > Micah > > -- > Ticket URL: > CM3 > Critical Mass Modula3 Compiler > > > ----- End forwarded message ----- > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 9 16:36:18 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 15:36:18 +0000 Subject: [M3devel] 386?486?586?686?etc.? In-Reply-To: References: , , <20100208035847.GB13743@topoi.pooq.com>, Message-ID: > I'm still running an old 100 MHz Pentium and using it on a daily basis. There was a recent thread on the gcc lists about this. Fedora 11 requires Pentium ("586"). Fedora 12 requires Pentium Pro ("686"). Various distros also use xz/lzma instead of gzip. But OpenBSD sticks with gzip for perf on old machines. Our .deb files use lzma, if the machine building them has the tools. >> Wow. What for? And with Modula-3? What OS? - Jay From: jay.krell at cornell.edu To: hendrik at topoi.pooq.com; m3devel at elegosoft.com Date: Mon, 8 Feb 2010 15:22:21 +0000 Subject: Re: [M3devel] 386?486?586?686?etc.? Wow. What for? And with Modula-3? What OS? I think Pentium will be ok. I think ultimately, if people really need, we should have separate targets. As I've been saying, like: I386_FREEBSD_USERTHREADS, I586_FREEBSD, etc. Esp. to enable easier "release engineering", such as when we do more cross builds, adding new targets will be cheaper. But we'd want some sort of system where if nobody downloads and installs and minimally tests a release, it is in some low grade classification. Certain ones we'd test automatically, like whatever we have available in Hudson. -Jay > Date: Sun, 7 Feb 2010 22:58:48 -0500 > From: hendrik at topoi.pooq.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] 386?486?586?686?etc.? > > On Sun, Feb 07, 2010 at 11:59:11PM +0000, Jay K wrote: > > > > Any opinions/counter-opinions on which processors we should support? > > > > Presumably it doesn't vary per platform. > > > > Like, it wouldn't be Linux/586 and FreeBSD/486 or such. > > > > Unless maybe there is clear data about what the kernels support? > > > > The atomic stuff is pushing things to i586. > > I believe before 486 and possibly 386 worked. > > > > 686 is probably reasonable. > > > > I think it is Pentium II or Pentium Pro or newer, stuff like 15 years old already. > > I'm still running an old 100 MHz Pentium and using it on a daily basis. > > Debian has dropped support for the 386 with, as far as I know, no > complaints. > > -- hendrik. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 9 16:53:25 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 15:53:25 +0000 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT In-Reply-To: References: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com>, Message-ID: Sadly the .msis don't appear on the download page. Perhaps they are there but not indexed? Not that I can tell. There are some here: http://modula3.elegosoft.com/cm3/uploaded-archives/ I'll try them out and make some new ones, from the release branch. Last I tried, and probably currently, there is an unfortunate strict matching requirement between our build and what version of Visual C++ you use. Therefore it behooves us to e.g. provide Visual C++ 8.0 and 9.0 builds. I suggested we double up the Hudson NT386 tasks in that way. I can upload pairs (or more) of .msis, but we are sort of supposed to not take "random developer builds" but use "official package builds". I find the fact that we produce 10+ packages per platform..confusing, overwhelming. I wish there was just one or perhaps two, even though they'd be large. I understand people like to factor things into small pieces, but monolithism has its upsides. - Jay From: jay.krell at cornell.edu To: wagner at elegosoft.com; m3devel at elegosoft.com Date: Tue, 9 Mar 2010 15:29:58 +0000 Subject: Re: [M3devel] Fwd: [CM3] #1082: Windows NT Try the .msi instead. Try this maybe, I can't find the link on the web page (maybe lost in the crash?) http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/www/installation-windows.html?rev=1.5;content-type=text%2Fplain I should edit/rewrite those. - The bit about "upgrade is pessmistic" isn't true, as my Russian friend says, it is "realistic". There are several breaking changes that motivate doing it the way upgrade does it. - I should steer people to Python instead of cmd (really). - Mentions of 5.2.6 and 5.5.0 and maybe even building from source should probably go away. Leave this for "more beginners". - I need to update the Visual C++ redistributable link (9.0SP1 vs. 9.0). - The config file path is wrong. It is not config-no-install instead of config. - It alludes to too many options, using various versions, building from source or not, etc. But it seems not too bad. It should be retested. The IDE is not what you would normally call an IDE. It is custom web server. Seems like "apples and oranges", but it does make some sense. In some ways it was ahead of its time -- people talk about "application servers" these days, and run C# in the web server to produce html. Just not clear what sorts of apps are suited to this model. There's no editor -- not a bad idea, since everyone prefers what they already have. There's no debugger -- not a bad idea, not like we have the time to write another. You can use Visual Studio or windbg. There is a bit of a project system. However Modula-3 has among the best text file driven command line build systems around, we just need it to handle walking directories, and have it determine the order. It does provide hyperlinking built source, though personally I just use my editor's "find in files" all the time. Maybe someone else can write up something to replace. http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/www/installation-windows.html?rev=1.5;content-type=text%2Fplain#buildm3current Fixing the .cmd to find the .exes should be trivial. Or just have user run the .exes. Have the .exes set any environment variables they need. - Jay > Date: Tue, 9 Mar 2010 15:25:48 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: [M3devel] Fwd: [CM3] #1082: Windows NT > > some windows problems again, probably only help needed... > > any takers? > > Olaf > > ----- Forwarded message from bugs at elego.de ----- > Date: Tue, 09 Mar 2010 02:55:44 -0000 > From: CM3 > Reply-To: CM3 > Subject: [CM3] #1082: Windows NT > To: @MISSING_DOMAIN > > #1082: Windows NT > -------------------------------------+-------------------------------------- > Reporter: ilikesci@? | Owner: wagner > Type: sw-test | Status: new > Priority: medium | Milestone: > Component: misc | Version: 5.8-RC3 > Severity: non-critical | Keywords: > Relnote: | Org: > Estimatedhours: 0 | Hours: 0 > Billable: 0 | Totalhours: 0 > Internal: 0 | > -------------------------------------+-------------------------------------- > Htr: > Remove msys out of your path. > > > Fix: > > > > Env: > Microsoft Windows Vista > > -------------------------------------+-------------------------------------- > I am new to modula-3 but have tried a few times to get cm3 working on my > MS-Windows machines over the years. This is my new attempt at that. I > downloaded the NT386 files and have them unpacked. I have put cm3 in > e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is > that when I ran cminstall it asked for a tar.exe, gzip.exe, and > msys-1.0.dll. I just happened to have msys installed on my computer and > dropped them in the folder and cminstall seemed to work. I just wanted to > let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and > so I ran it per the suggestion on your website. It asked a few questions > about what browser I wanted to use and such. The browser does pop up on > localhost:3800 but times out. The console spits out: > > Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 > calling start_browser(http://localhost:3800/) > starting TCP service > start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" > http://localhost:3800/ > CM3-IDE is shutting down because start_browser() returned TRUE. > TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept > TCPServer: aborting... > > I am thinking maybe something else might be running on the port and > causing a problem loading page. I copied the startReactor.cmd to the bin > directory and tried it and got the following: > > > ------------------------------------------------------------------------------- > startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by > RCC > > ------------------------------------------------------------------------------- > FATAL ERROR: Unable to find CM3 installation. > CM3_ROOT expected in folder C:\cm3 > CM3_BIN expected in folder C:\cm3\bin > CM3.EXE expected in file C:\cm3\bin\cm3.exe > Reactor.EXE expected in file C:\cm3\bin\reactor.exe > cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD > > ------------------------------------------------------------------------------- > Which it is installed on e: instead of c: but there is not a reactor.exe > file in the bin directory either. From that information I am thinking I > should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as > e:\cm3. > > This is FYI and will let you know if I continue and any other experiences > that might be of use. > > Thank You, > Micah > > -- > Ticket URL: > CM3 > Critical Mass Modula3 Compiler > > > ----- End forwarded message ----- > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 9 17:46:52 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 16:46:52 +0000 Subject: [M3devel] PPC64_DARWIN Message-ID: Fyi, PPC64_DARWIN mostly works. "Ship" has a strong tendency to hang. Whenever I've looked, the stack has no symbols. I tried growing jmpbuf, using default stack, no luck yet. Will look more "later". Anyone running a Mac/G5? It's a big endian 64bit platform, if that is interesting. :) (SPARC64_SOLARIS will be too.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Tue Mar 9 15:43:44 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Tue, 9 Mar 2010 09:43:44 -0500 Subject: [M3devel] 386?486?586?686?etc.? In-Reply-To: References: Message-ID: <20100309144344.GC12734@topoi.pooq.com> On Tue, Mar 09, 2010 at 03:36:18PM +0000, Jay K wrote: > > > I'm still running an old 100 MHz Pentium and using it on a daily basis. > >> Wow. What for? And with Modula-3? What OS? It's my most reliable machine. The kind of reliable that's been running for something like two decades and shows signs of running for two more if I can continue to get secure OS support for it. My newer machines keep burning up graphics cards and the like. It is running Debian Lenny at the moment; a few months ago it was on Debian Etch. It's the boundary between my LAN and the internet. I last used Modula 3 on it a few years ago, throwing together an experimental type-checked language interpreter to test some ideas about language design. I'm not running Modula 3 on it right now, but since Modula 3 is still my systems language of choice, I'd hesitate to say I won't be doing it again. - hendrik > > > - Jay > > > > > > From: jay.krell at cornell.edu > To: hendrik at topoi.pooq.com; m3devel at elegosoft.com > Date: Mon, 8 Feb 2010 15:22:21 +0000 > Subject: Re: [M3devel] 386?486?586?686?etc.? > > > > Wow. What for? And with Modula-3? What OS? > I think Pentium will be ok. > I think ultimately, if people really need, we should have separate targets. > As I've been saying, like: I386_FREEBSD_USERTHREADS, I586_FREEBSD, etc. > Esp. to enable easier "release engineering", such as when we do more cross builds, > adding new targets will be cheaper. But we'd want some sort of system > where if nobody downloads and installs and minimally tests a release, it is > in some low grade classification. > Certain ones we'd test automatically, like whatever we have available in Hudson. > > -Jay > > > Date: Sun, 7 Feb 2010 22:58:48 -0500 > > From: hendrik at topoi.pooq.com > > To: m3devel at elegosoft.com > > Subject: Re: [M3devel] 386?486?586?686?etc.? > > > > On Sun, Feb 07, 2010 at 11:59:11PM +0000, Jay K wrote: > > > > > > Any opinions/counter-opinions on which processors we should support? > > > > > > Presumably it doesn't vary per platform. > > > > > > Like, it wouldn't be Linux/586 and FreeBSD/486 or such. > > > > > > Unless maybe there is clear data about what the kernels support? > > > > > > The atomic stuff is pushing things to i586. > > > I believe before 486 and possibly 386 worked. > > > > > > 686 is probably reasonable. > > > > > > I think it is Pentium II or Pentium Pro or newer, stuff like 15 years old already. > > > > I'm still running an old 100 MHz Pentium and using it on a daily basis. > > > > Debian has dropped support for the 386 with, as far as I know, no > > complaints. > > > > -- hendrik. > From rcolebur at SCIRES.COM Wed Mar 10 03:10:57 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Tue, 9 Mar 2010 21:10:57 -0500 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT In-Reply-To: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com> References: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com> Message-ID: Olaf: The "startReactor.CMD" file is obsolete and no longer in the repository. The new files are in "scripts\install\windows" and should be copied to the "cm3\bin" folder on a windows installation. "cm3StartIDE.CMD" replaces "startReactor.CMD". This file makes use of "cm3CommandShell.CMD". It appears that the problem reporter has his installation rooted at E:\cm3 rather than C:\cm3. So, he will need to set an environment variable "set CM3_ROOT=E:\cm3" to let these scripts know where to find the cm3 installation. Alternately, cm3CommandShell can be invoked with the arguments "Root E:\cm3" to tell it where to find the installation. Regards, Randy -----Original Message----- From: Olaf Wagner [mailto:wagner at elegosoft.com] Sent: Tuesday, March 09, 2010 9:26 AM To: m3devel Subject: [M3devel] Fwd: [CM3] #1082: Windows NT some windows problems again, probably only help needed... any takers? Olaf ----- Forwarded message from bugs at elego.de ----- Date: Tue, 09 Mar 2010 02:55:44 -0000 From: CM3 Reply-To: CM3 Subject: [CM3] #1082: Windows NT To: @MISSING_DOMAIN #1082: Windows NT -------------------------------------+-------------------------------------- Reporter: ilikesci@? | Owner: wagner Type: sw-test | Status: new Priority: medium | Milestone: Component: misc | Version: 5.8-RC3 Severity: non-critical | Keywords: Relnote: | Org: Estimatedhours: 0 | Hours: 0 Billable: 0 | Totalhours: 0 Internal: 0 | -------------------------------------+-------------------------------------- Htr: Remove msys out of your path. Fix: Env: Microsoft Windows Vista -------------------------------------+-------------------------------------- I am new to modula-3 but have tried a few times to get cm3 working on my MS-Windows machines over the years. This is my new attempt at that. I downloaded the NT386 files and have them unpacked. I have put cm3 in e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is that when I ran cminstall it asked for a tar.exe, gzip.exe, and msys-1.0.dll. I just happened to have msys installed on my computer and dropped them in the folder and cminstall seemed to work. I just wanted to let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and so I ran it per the suggestion on your website. It asked a few questions about what browser I wanted to use and such. The browser does pop up on localhost:3800 but times out. The console spits out: Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 calling start_browser(http://localhost:3800/) starting TCP service start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" http://localhost:3800/ CM3-IDE is shutting down because start_browser() returned TRUE. TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept TCPServer: aborting... I am thinking maybe something else might be running on the port and causing a problem loading page. I copied the startReactor.cmd to the bin directory and tried it and got the following: ------------------------------------------------------------------------------- startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by RCC ------------------------------------------------------------------------------- FATAL ERROR: Unable to find CM3 installation. CM3_ROOT expected in folder C:\cm3 CM3_BIN expected in folder C:\cm3\bin CM3.EXE expected in file C:\cm3\bin\cm3.exe Reactor.EXE expected in file C:\cm3\bin\reactor.exe cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD ------------------------------------------------------------------------------- Which it is installed on e: instead of c: but there is not a reactor.exe file in the bin directory either. From that information I am thinking I should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as e:\cm3. This is FYI and will let you know if I continue and any other experiences that might be of use. Thank You, Micah -- Ticket URL: CM3 Critical Mass Modula3 Compiler ----- End forwarded message ----- -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. From jay.krell at cornell.edu Wed Mar 10 07:52:42 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 10 Mar 2010 06:52:42 +0000 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT In-Reply-To: References: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com>, Message-ID: %~dp0 is a cmd file's directory. If cmd and exe are next to each other, they can find each other. As well, what does the cmd do that it can't be done in the exe? - Jay (phone) > From: rcolebur at SCIRES.COM > To: wagner at elegosoft.com; m3devel at elegosoft.com > Date: Tue, 9 Mar 2010 21:10:57 -0500 > Subject: Re: [M3devel] Fwd: [CM3] #1082: Windows NT > > Olaf: > > The "startReactor.CMD" file is obsolete and no longer in the repository. > > The new files are in "scripts\install\windows" and should be copied to the "cm3\bin" folder on a windows installation. > > "cm3StartIDE.CMD" replaces "startReactor.CMD". This file makes use of "cm3CommandShell.CMD". > > It appears that the problem reporter has his installation rooted at E:\cm3 rather than C:\cm3. So, he will need to set an environment variable "set CM3_ROOT=E:\cm3" to let these scripts know where to find the cm3 installation. Alternately, cm3CommandShell can be invoked with the arguments "Root E:\cm3" to tell it where to find the installation. > > Regards, > Randy > > -----Original Message----- > From: Olaf Wagner [mailto:wagner at elegosoft.com] > Sent: Tuesday, March 09, 2010 9:26 AM > To: m3devel > Subject: [M3devel] Fwd: [CM3] #1082: Windows NT > > some windows problems again, probably only help needed... > > any takers? > > Olaf > > ----- Forwarded message from bugs at elego.de ----- > Date: Tue, 09 Mar 2010 02:55:44 -0000 > From: CM3 > Reply-To: CM3 > Subject: [CM3] #1082: Windows NT > To: @MISSING_DOMAIN > > #1082: Windows NT > -------------------------------------+-------------------------------------- > Reporter: ilikesci@? | Owner: wagner > Type: sw-test | Status: new > Priority: medium | Milestone: > Component: misc | Version: 5.8-RC3 > Severity: non-critical | Keywords: > Relnote: | Org: > Estimatedhours: 0 | Hours: 0 > Billable: 0 | Totalhours: 0 > Internal: 0 | > -------------------------------------+-------------------------------------- > Htr: > Remove msys out of your path. > > > Fix: > > > > Env: > Microsoft Windows Vista > > -------------------------------------+-------------------------------------- > I am new to modula-3 but have tried a few times to get cm3 working on my > MS-Windows machines over the years. This is my new attempt at that. I > downloaded the NT386 files and have them unpacked. I have put cm3 in > e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is > that when I ran cminstall it asked for a tar.exe, gzip.exe, and > msys-1.0.dll. I just happened to have msys installed on my computer and > dropped them in the folder and cminstall seemed to work. I just wanted to > let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and > so I ran it per the suggestion on your website. It asked a few questions > about what browser I wanted to use and such. The browser does pop up on > localhost:3800 but times out. The console spits out: > > Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 > calling start_browser(http://localhost:3800/) > starting TCP service > start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" > http://localhost:3800/ > CM3-IDE is shutting down because start_browser() returned TRUE. > TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept > TCPServer: aborting... > > I am thinking maybe something else might be running on the port and > causing a problem loading page. I copied the startReactor.cmd to the bin > directory and tried it and got the following: > > > ------------------------------------------------------------------------------- > startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by > RCC > > ------------------------------------------------------------------------------- > FATAL ERROR: Unable to find CM3 installation. > CM3_ROOT expected in folder C:\cm3 > CM3_BIN expected in folder C:\cm3\bin > CM3.EXE expected in file C:\cm3\bin\cm3.exe > Reactor.EXE expected in file C:\cm3\bin\reactor.exe > cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD > > ------------------------------------------------------------------------------- > Which it is installed on e: instead of c: but there is not a reactor.exe > file in the bin directory either. From that information I am thinking I > should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as > e:\cm3. > > This is FYI and will let you know if I continue and any other experiences > that might be of use. > > Thank You, > Micah > > -- > Ticket URL: > CM3 > Critical Mass Modula3 Compiler > > > ----- End forwarded message ----- > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > > > CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. > > EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Wed Mar 10 14:10:41 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Wed, 10 Mar 2010 14:10:41 +0100 Subject: [M3devel] [CM3] #1082: Windows NT Message-ID: <20100310141041.t3hp8s034ss0cggc@mail.elegosoft.com> Have you read the answers on the m3devel list to which I forwarded your report? If not, you will find them in the archives at https://mail.elegosoft.com/pipermail/m3devel/2010-March/thread.html Hope that helps, @Jay/Randy, you should add your information to the trac ticket, too: https://projects.elego.de/cm3/ticket/1082 Sorry that I cannot do much myself currently, Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From rcolebur at SCIRES.COM Wed Mar 10 16:35:29 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Wed, 10 Mar 2010 10:35:29 -0500 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT In-Reply-To: References: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com>, Message-ID: Jay, the CMD files are conveniences I've found useful under Windows. I can supply some .REG files that will allow you to integrate these into the Explorer shell so that you can start a cm3 command shell in any folder and you can start CM3IDE. These CMD files ensure the environment is set up properly, including Visual C++. I understand the use of %~dp0 , but I already have an optional command line argument to deal with different locations for cm3 installation. Nevertheless, I can build in some more search capability in the cmd files if desired. Perhaps using %~dp0 would be a good tactic; I'll look into adding it. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, March 10, 2010 1:53 AM To: Coleburn, Randy; Olaf Wagner; m3devel Subject: RE: [M3devel] Fwd: [CM3] #1082: Windows NT %~dp0 is a cmd file's directory. If cmd and exe are next to each other, they can find each other. As well, what does the cmd do that it can't be done in the exe? - Jay (phone) > From: rcolebur at SCIRES.COM > To: wagner at elegosoft.com; m3devel at elegosoft.com > Date: Tue, 9 Mar 2010 21:10:57 -0500 > Subject: Re: [M3devel] Fwd: [CM3] #1082: Windows NT > > Olaf: > > The "startReactor.CMD" file is obsolete and no longer in the repository. > > The new files are in "scripts\install\windows" and should be copied to the "cm3\bin" folder on a windows installation. > > "cm3StartIDE.CMD" replaces "startReactor.CMD". This file makes use of "cm3CommandShell.CMD". > > It appears that the problem reporter has his installation rooted at E:\cm3 rather than C:\cm3. So, he will need to set an environment variable "set CM3_ROOT=E:\cm3" to let these scripts know where to find the cm3 installation. Alternately, cm3CommandShell can be invoked with the arguments "Root E:\cm3" to tell it where to find the installation. > > Regards, > Randy > > -----Original Message----- > From: Olaf Wagner [mailto:wagner at elegosoft.com] > Sent: Tuesday, March 09, 2010 9:26 AM > To: m3devel > Subject: [M3devel] Fwd: [CM3] #1082: Windows NT > > some windows problems again, probably only help needed... > > any takers? > > Olaf > > ----- Forwarded message from bugs at elego.de ----- > Date: Tue, 09 Mar 2010 02:55:44 -0000 > From: CM3 > Reply-To: CM3 > Subject: [CM3] #1082: Windows NT > To: @MISSING_DOMAIN > > #1082: Windows NT > -------------------------------------+-------------------------------------- > Reporter: ilikesci at ... | Owner: wagner > Type: sw-test | Status: new > Priority: medium | Milestone: > Component: misc | Version: 5.8-RC3 > Severity: non-critical | Keywords: > Relnote: | Org: > Estimatedhours: 0 | Hours: 0 > Billable: 0 | Totalhours: 0 > Internal: 0 | > -------------------------------------+-------------------------------------- > Htr: > Remove msys out of your path. > > > Fix: > > > > Env: > Microsoft Windows Vista > > -------------------------------------+-------------------------------------- > I am new to modula-3 but have tried a few times to get cm3 working on my > MS-Windows machines over the years. This is my new attempt at that. I > downloaded the NT386 files and have them unpacked. I have put cm3 in > e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is > that when I ran cminstall it asked for a tar.exe, gzip.exe, and > msys-1.0.dll. I just happened to have msys installed on my computer and > dropped them in the folder and cminstall seemed to work. I just wanted to > let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and > so I ran it per the suggestion on your website. It asked a few questions > about what browser I wanted to use and such. The browser does pop up on > localhost:3800 but times out. The console spits out: > > Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 > calling start_browser(http://localhost:3800/) > starting TCP service > start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" > http://localhost:3800/ > CM3-IDE is shutting down because start_browser() returned TRUE. > TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept > TCPServer: aborting... > > I am thinking maybe something else might be running on the port and > causing a problem loading page. I copied the startReactor.cmd to the bin > directory and tried it and got the following: > > > ------------------------------------------------------------------------------- > startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by > RCC > > ------------------------------------------------------------------------------- > FATAL ERROR: Unable to find CM3 installation. > CM3_ROOT expected in folder C:\cm3 > CM3_BIN expected in folder C:\cm3\bin > CM3.EXE expected in file C:\cm3\bin\cm3.exe > Reactor.EXE expected in file C:\cm3\bin\reactor.exe > cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD > > ------------------------------------------------------------------------------- > Which it is installed on e: instead of c: but there is not a reactor.exe > file in the bin directory either. From that information I am thinking I > should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as > e:\cm3. > > This is FYI and will let you know if I continue and any other experiences > that might be of use. > > Thank You, > Micah > > -- > Ticket URL: > CM3 > Critical Mass Modula3 Compiler > > > ----- End forwarded message ----- > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > > > CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. > > EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 10 16:56:32 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 10 Mar 2010 15:56:32 +0000 Subject: [M3devel] NT386 64bit LONGINT sub range checks oops Message-ID: Just a note that subrange checking of 64bit LONGINT is broken on NT386. I just noticed. 64bit LONGINT should work to a very large extent though. This is the only problem I know of. I'll look into it soon. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 10 17:09:46 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 10 Mar 2010 16:09:46 +0000 Subject: [M3devel] NT386 64bit LONGINT sub range checks oops In-Reply-To: References: Message-ID: Hm. I'm just not sure actually. I'll have to test it. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Wed, 10 Mar 2010 15:56:32 +0000 Subject: [M3devel] NT386 64bit LONGINT sub range checks oops Just a note that subrange checking of 64bit LONGINT is broken on NT386. I just noticed. 64bit LONGINT should work to a very large extent though. This is the only problem I know of. I'll look into it soon. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 11 15:45:21 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 11 Mar 2010 14:45:21 +0000 Subject: [M3devel] interface long, inlining large integer code, etc. In-Reply-To: <60FB82E7-B3B3-45ED-973B-191F7992A4E2@cs.purdue.edu> References: <20100308122636.83CA12474001@birch.elegosoft.com>, , , , , , , , , , , , , <5ECDB225-60F1-4094-96EA-B7BB0EBE340A@cs.purdue.edu>, , <60FB82E7-B3B3-45ED-973B-191F7992A4E2@cs.purdue.edu> Message-ID: I was thinking a bit more about this. Basically it can already be done -- we know the name of the function are compiling. If the function is Long__Extract, Long__Times, etc., we can inline. If it isn't, we can assume the existance and interface of such functions, and call them. Instead of calling anything in hand.c. Maybe even give them custom calling conventions. The front end knows stuff about interface Word/Long. It seems reasonable for m3back to also. The open question however is that there is no place to "hang" inlined versions of signed operations, what you might call Integer__Extract, Integer__Divide, LongInt__Div, LongInt__Times, LongInt__Mod, etc. Maybe they should be in C:\dev2\cm3.2\m3-libs\m3core\src\types? Similarly open question would be the set functions. RTHooks.SetLT? RTSet.LT? C:\dev2\cm3.2\m3-libs\m3core\src\types\set? C:\dev2\cm3.2\m3-libs\m3core\src\types\bigset? (as opposed to word-sized set, besides that eq/ne are inlined by the frontend for "medium" sizes) Is it clear what my questions are? Let me try again. Consider Long.Rotate. Assume it takes a bunch of code. m3back could notice it is compiling Long__Rotate, and generate the code inline. Otherwise it can "know" such a function exists and generate a call to it. This is a reasonable seeming model that I just came up with. That does double the paths in m3back, granted. For several such m3cg operations, there is already a natural function to treat this way -- stuff in interface Long. However, for a few operations, there is not already a place -- signed operations and operations on large sets. As well, there is a presumed question/answer/non-answer: everything works today. But should it be changed? Is it better to reduce/eliminate hand.c and instead teach the Modula-3 code in m3back how to generate assembly? "Better" how? More efficient? Maybe. The factor here is that hand.c may or may not be optimized, but m3back operates in a mode where it always optimizes to some extent. It generally generates code that is significantly better than unoptimized C, and significantly worse than optimized C, depending -- it does pretty well, but it never inlines, but it only does a certain small class of smart things. It keeps values in registers somewhat, it doesn't eliminate dead stores, doesn't unroll loops. Another factor is the principle of having C code or not. We vehemently agree that C code is preferred in m3core/unix. But otherwise many people here kind of dislike or discourage C on principle/philosophical grounds. I'm not sure I agree, however by agreeing, I introduce an interesting technical challenge. :) Again, must we really call it interface long? Does that really "sound like": INTEGER is to Word as LONGINT is to x? LongWord maybe? - Jay > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > CC: m3commit at elegosoft.com > Subject: RE: [M3commit] CVS Update: cm3 > Date: Thu, 11 Mar 2010 04:10:40 +0000 > > > Long.LeftShift is not a great example. Sorry. > Long.Shift, Long.Rotate, Long.Times, LongDivide. > Those are or "could be" large. Currently the later > three all call functions and are therefore short. > > > Integer.Multiple, Divide, Remainder, similarly uncertain. > They call functions, the functions aren't short. > > > I was also thinking, some of these functions, esp. > 64bit shift/rotate could be shorter as loops. > But I think not worthwhile. > > > To be clear, even though "parse.c" "never" generates > function calls (except for set operations), the gcc > backend itself often does. The functions are in libgcc. > > > You know..I find it hard to decide about compiler generated > function calls. Definitely for size optimization they > can be a win. But I just somehow vaguely don't like them. > In many cases, the decision is not difficult. set_singleton > and set_member are good examples. They could be inlined > and be very small. But Shift, Rotate, Multiply, Divide, > it gets less clear. > > > But no matter how large an inline form, I think > when interface word/long maps directly to these operations, > they should be inlined. > > > And then if there are places that don't inline, they should > call the "instantiations" in interface word/long. > > > And then, furthermore, we should consider an interface > for "Integer.Divide/Multiple/Remainder"? > A place to park the possibly large code? > > > - Jay > > > ---------------------------------------- > > From: hosking at cs.purdue.edu > > Date: Wed, 10 Mar 2010 21:01:54 -0500 > > To: jay.krell at cornell.edu > > CC: m3commit at elegosoft.com > > Subject: Re: [M3commit] CVS Update: cm3 > > > > I am confused. The intention is that Long.LeftShift is always inlined wherever it is called. Is the code for it really that hairy? > > > > On 10 Mar 2010, at 20:30, Jay K wrote: > > > >> > >> If there exists an inline expansion, and it is large, but there is a function whose job it is to hold the expansion, whether it is inline or not, and the backend was informed of this function's name, and whether or not it was currently producing it, the backend could generate the inline expansion for Long__LeftShift, but otherwise generate calls to it. > >> > >> > >> That is, e.g. m3cg.left_shift(type := int64|word64) could chose to either call Long__LeftShift or generate the body of Long__LeftShift. Not based on some "is inlining profitable heuristic", but specifically if told it is generating the function Long__LeftShift or not. > >> > >> > >> That is, there's no point in having a C function m3_left_shift64 or somesuch, and having Long__LeftShift call it. Instead a backend should generate the body of Long__LeftShift inline when told it is generating that function, vs. generating a call to that function when it is otherwise asked to do a 64bit leftshift if it decides that inlining it every time is too large. > >> > >> Granted at least two things: > >> Given 64bit target, Long__LeftShift vs. Word__LeftShift is ambiguous. > >> And I'm inlined to just always inline anyway. > >> > >> > >> (I still don't like the term "Long" here. It doesn't convey unsigned.) > >> > >> > >> - Jay > >> > >> > >> > >> ________________________________ > >>> Subject: Re: [M3commit] CVS Update: cm3 > >>> From: hosking at cs.purdue.edu > >>> Date: Wed, 10 Mar 2010 19:57:17 -0500 > >>> CC: m3commit at elegosoft.com > >>> To: jay.krell at cornell.edu > >>> > >>> > >>> > >>> Yes, someone can pass the function as a parameter. > >>> > >>> I don't understand the rest of what you are saying. > >>> > >>> > >>> Antony Hosking | Associate Professor | Computer Science | Purdue University > >>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>> > >>> > >>> > >>> > >>> > >>> > >>> On 10 Mar 2010, at 17:27, Jay K wrote: > >>> > >>> > >>> Hey, should we maybe have a flag that indicates we are producing a function that is meant to provide and only provide exactly the functionality of the "one" m3cg call? > >>> You know, that'd be a great hint to: > >>> > >>> - definitely definitely definitely definitely inline > >>> - if we know this thing existed, we could use it instead of a .c file. > >>> - along with the function's name for the invocations that aren't the function? > >>> > >>> > >>> Why provide the function anyway? In case someone takes the address? > >>> > >>> > >>> > >>> - Jay > >>> > >>> > >>> ________________________________ > >>> From: jay.krell at cornell.edu > >>> To: hosking at cs.purdue.edu > >>> Date: Mon, 8 Mar 2010 16:04:45 +0000 > >>> CC: m3commit at elegosoft.com > >>> Subject: Re: [M3commit] CVS Update: cm3 > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> Fair enough. > >>> > >>> > >>> > >>> - Jay > >>> > >>> > >>> > >>> ________________________________ > >>> > >>> From: hosking at cs.purdue.edu > >>> Date: Mon, 8 Mar 2010 10:29:00 -0500 > >>> To: jay.krell at cornell.edu > >>> CC: m3commit at elegosoft.com > >>> Subject: Re: [M3commit] CVS Update: cm3 > >>> > >>> > >>> > >>> > >>> You can't call the support function Long__Extract because we already have a Long__Extract in m3core/src/word. It is the library representative of the front-end compiler-generated (inlined) Long.Extract. > >>> > >>> > >>> > >>> On 8 Mar 2010, at 07:30, Jay K wrote: > >>> > >>> > >>> > >>> diff attached > >>> > >>> > >>> If folks really want to use tables or function calls here, let me know. > >>> > >>> > >>> The data was historically problematic, though we worked out the problems. > >>> - It was initialized at runtime which has a race condition, fixed years ago. > >>> - It can't be "exported" on NT386 (which is the only platform that uses it) and must be statically linked. > >>> > >>> The data is still used by Word.Insert and Long.Insert is still a function, but those > >>> are my next targets. > >>> > >>> > >>> This is a case where the user does write a function call. Word.Extract or Long.Extract. > >>> (So the function should have been called Long__Extract.) > >>> > >>> > >>> - Jay > >>> > >>> Date: Mon, 8 Mar 2010 13:26:36 +0000 > >>> To: m3commit at elegosoft.com > >>> From: jkrell at elego.de > >>> Subject: [M3commit] CVS Update: cm3 > >>> > >>> CVSROOT: /usr/cvs > >>> Changes by: jkrell at birch. 10/03/08 13:26:36 > >>> > >>> Modified files: > >>> cm3/m3-sys/m3back/src/: M3x86.m3 Stackx86.i3 Stackx86.m3 > >>> > >>> Log message: > >>> "rewrite" extract to not use tables and always be inlined for 64bit > >>> > >>> equivalent C code: > >>> > >>> UT __stdcall extract(UT x, uint32 offset, uint32 count) > >>> { > >>> x>>= offset; > >>> x &= ~((~(UT)0) << count); > >>> return x; > >>> } > >>> > >>> extract32 > >>> 00000729: 8B4DEC MOV ECX tv.126[_m] > >>> 0000072C: 8B5DF4 MOV EBX tv.124[_a32] > >>> 0000072F: D3EB SHR EBX > >>> 00000731: 8BD1 MOV EDX ECX This is pointless and I'll try to remove. > >>> 00000733: 8B4DE4 MOV ECX tv.128[_n] > >>> 00000736: BEFFFFFFFF MOV ESI $4294967295 I'll look for smaller form of this. or esi -1? > >>> 0000073B: D3E6 SHL ESI > >>> 0000073D: F7D6 NOT ESI > >>> 0000073F: 23DE AND EBX ESI > >>> > >>> extract64 > >>> 000008E4: 8B4DD8 MOV ECX tv.134[_m] > >>> 000008E7: 8B5DE8 MOV EBX tv.131[_a64] > >>> 000008EA: 8B55EC MOV EDX tv.131[_a64]+4 > >>> 000008ED: 0FADD3 SHRD EBX EDX ECX > >>> 000008F0: D3EA SHR EDX > >>> 000008F2: F6C120 TEST ECX $32 > >>> 000008F5: 7400 JE rel8 L.107 > >>> 000008F7: 8BDA MOV EBX EDX > >>> 000008F9: 33D2 XOR EDX EDX > >>> set_label L.107 > >>> 000008FB: 8BF1 MOV ESI ECX This is pointless and I'll try to remove. > >>> 000008FD: 8B4DD0 MOV ECX tv.136[_n] > >>> 00000900: BFFFFFFFFF MOV EDI $4294967295 I'll look for smaller form of this. or edi -1? > >>> 00000905: B8FFFFFFFF MOV EAX $4294967295 I'll look for smaller form of this. (heck, at least mov edi, eax) > >>> 0000090A: 0FA5F8 SHLD EAX EDI ECX > >>> 0000090D: D3E7 SHL EDI > >>> 0000090F: F6C120 TEST ECX $32 > >>> 00000912: 7400 JE rel8 L.108 > >>> 00000914: 8BC7 MOV EAX EDI > >>> 00000916: 33FF XOR EDI EDI > >>> set_label L.108 > >>> 00000918: F7D7 NOT EDI > >>> 0000091A: F7D0 NOT EAX > >>> 0000091C: 23DF AND EBX EDI > >>> 0000091E: 23D0 AND EDX EAX > >>> > >>> having n or m and n (or just m? I think so.) be constant leads to better code > >>> > >>> some other small cleanup, like avoiding calling find twice, > >>> I don't see why it was that way > >>> > >>> <1.txt> > >>> > >>> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 11 16:40:34 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 11 Mar 2010 15:40:34 +0000 Subject: [M3devel] Target.EOL Message-ID: I strongly suspect that Target.EOL can go away and just use Wr.EOL instead. Or even just \n. Cross builds are fairly rare, esp. cross between NT and Posix, and very many tools treat \n, \r\n, and perhaps \r the same, so even crossing NT <=> Posix doesn't likely matter. ie. it works, assuming you have a cross m3cg (ie: mingwin/cygwin hosted for NT hosts) gcc, Visual C++ compiler, m3front, all treat \r and \r\n the same. (witness all our *.h, *.c, *.i3, *.m3 files use \r\n) I think we have some tools that don't understand \r\n, but in my opinion that is a bug. All three formats should be treated the same. I know notepad doesn't display \n well, and Sun cc might not like \r\n. I know some compiler I used recently was picky, but "many" compilers are not. cmd might be ok with \n. Python calls it "universal newlines", treating all three formats the same. C:\dev2\cm3.2\m3-sys\cm3\src\Utils.m3(190):PROCEDURE CopyText (old, new: TEXT) = C:\dev2\cm3.2\m3-sys\m3middle\src\M3File.m3(61):PROCEDURE CopyText (src, dest: TEXT; eol: TEXT) RAISES {OSError.E} = probably leave alone, except that they don't treat \r correctly if you consider the historical Apple behavior. There are three formats, not two. Anyway, it's not a big deal. I think my problem here is solved by the change to m3x86.m3 I already made. I'll put Target.EOL back on my machine. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 11 18:31:42 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 11 Mar 2010 12:31:42 -0500 Subject: [M3devel] Target.EOL In-Reply-To: References: Message-ID: <6A2F1518-1C9C-4666-BC3E-B6B8D397A9EA@cs.purdue.edu> Sounds reasonable. On 11 Mar 2010, at 10:40, Jay K wrote: > I strongly suspect that Target.EOL can go away and just use Wr.EOL instead. > Or even just \n. > > > Cross builds are fairly rare, esp. cross between NT and Posix, and very > many tools treat \n, \r\n, and perhaps \r the same, so even > crossing NT <=> Posix doesn't likely matter. > ie. it works, assuming you have a cross m3cg (ie: mingwin/cygwin hosted for NT hosts) > > > gcc, Visual C++ compiler, m3front, all treat \r and \r\n the same. > (witness all our *.h, *.c, *.i3, *.m3 files use \r\n) > I think we have some tools that don't understand \r\n, but in my opinion that is a bug. > All three formats should be treated the same. > > > I know notepad doesn't display \n well, and Sun cc might not like \r\n. > I know some compiler I used recently was picky, but "many" compilers are not. > cmd might be ok with \n. > Python calls it "universal newlines", treating all three formats the same. > > > C:\dev2\cm3.2\m3-sys\cm3\src\Utils.m3(190):PROCEDURE CopyText (old, new: TEXT) = > C:\dev2\cm3.2\m3-sys\m3middle\src\M3File.m3(61):PROCEDURE CopyText (src, dest: TEXT; eol: TEXT) RAISES {OSError.E} = > > > probably leave alone, except that they don't treat \r correctly if you consider the historical Apple behavior. > There are three formats, not two. > > > Anyway, it's not a big deal. I think my problem here is solved by the change to m3x86.m3 I already made. > I'll put Target.EOL back on my machine. > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 11 19:03:43 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 11 Mar 2010 13:03:43 -0500 Subject: [M3devel] interface long, inlining large integer code, etc. In-Reply-To: References: <20100308122636.83CA12474001@birch.elegosoft.com>, , , , , , , , , , , , , <5ECDB225-60F1-4094-96EA-B7BB0EBE340A@cs.purdue.edu>, , <60FB82E7-B3B3-45ED-973B-191F7992A4E2@cs.purdue.edu> Message-ID: <8536088B-FFE4-43CA-AC63-442D66AE72A0@cs.purdue.edu> I'm going to resist this one. It's not clear that it matters terribly much whether these utility routines are inlined or not. If it is shown to be a performance bottleneck then perhaps. But I suspect that there are very few programs that pass the procedures of the Word/Long interface around by value. On 11 Mar 2010, at 09:45, Jay K wrote: > I was thinking a bit more about this. > Basically it can already be done -- we know the name of the function are compiling. > If the function is Long__Extract, Long__Times, etc., we can inline. > If it isn't, we can assume the existance and interface of such functions, and call them. > Instead of calling anything in hand.c. > Maybe even give them custom calling conventions. > The front end knows stuff about interface Word/Long. It seems reasonable for m3back to also. > > > The open question however is that there is no place to "hang" inlined versions of signed operations, what you might call Integer__Extract, Integer__Divide, LongInt__Div, LongInt__Times, LongInt__Mod, etc. > > > Maybe they should be in C:\dev2\cm3.2\m3-libs\m3core\src\types? > > > Similarly open question would be the set functions. > RTHooks.SetLT? > RTSet.LT? > C:\dev2\cm3.2\m3-libs\m3core\src\types\set? > C:\dev2\cm3.2\m3-libs\m3core\src\types\bigset? > (as opposed to word-sized set, besides that eq/ne are inlined by the frontend for "medium" sizes) > > > Is it clear what my questions are? > > > Let me try again. > Consider Long.Rotate. Assume it takes a bunch of code. > m3back could notice it is compiling Long__Rotate, and generate the code inline. > Otherwise it can "know" such a function exists and generate a call to it. > This is a reasonable seeming model that I just came up with. > > > That does double the paths in m3back, granted. > > For several such m3cg operations, there is already a natural function to treat this way -- stuff in interface Long. > > > However, for a few operations, there is not already a place -- signed operations and operations on large sets. > > > As well, there is a presumed question/answer/non-answer: everything works today. > But should it be changed? > Is it better to reduce/eliminate hand.c and instead teach the Modula-3 code in m3back how to generate assembly? > "Better" how? > More efficient? Maybe. The factor here is that hand.c may or may not be optimized, but m3back operates in a mode where it always optimizes to some extent. It generally generates code that is significantly better than unoptimized C, and significantly worse than optimized C, depending -- it does pretty well, but it never inlines, but it only does a certain small class of smart things. It keeps values in registers somewhat, it doesn't eliminate dead stores, doesn't unroll loops. > > > Another factor is the principle of having C code or not. > We vehemently agree that C code is preferred in m3core/unix. > But otherwise many people here kind of dislike or discourage C on principle/philosophical grounds. > I'm not sure I agree, however by agreeing, I introduce an interesting technical challenge. :) > > > Again, must we really call it interface long? > Does that really "sound like": > INTEGER is to Word as LONGINT is to x? > > LongWord maybe? > > > - Jay > > > > From: jay.krell at cornell.edu > > To: hosking at cs.purdue.edu > > CC: m3commit at elegosoft.com > > Subject: RE: [M3commit] CVS Update: cm3 > > Date: Thu, 11 Mar 2010 04:10:40 +0000 > > > > > > Long.LeftShift is not a great example. Sorry. > > Long.Shift, Long.Rotate, Long.Times, LongDivide. > > Those are or "could be" large. Currently the later > > three all call functions and are therefore short. > > > > > > Integer.Multiple, Divide, Remainder, similarly uncertain. > > They call functions, the functions aren't short. > > > > > > I was also thinking, some of these functions, esp. > > 64bit shift/rotate could be shorter as loops. > > But I think not worthwhile. > > > > > > To be clear, even though "parse.c" "never" generates > > function calls (except for set operations), the gcc > > backend itself often does. The functions are in libgcc. > > > > > > You know..I find it hard to decide about compiler generated > > function calls. Definitely for size optimization they > > can be a win. But I just somehow vaguely don't like them. > > In many cases, the decision is not difficult. set_singleton > > and set_member are good examples. They could be inlined > > and be very small. But Shift, Rotate, Multiply, Divide, > > it gets less clear. > > > > > > But no matter how large an inline form, I think > > when interface word/long maps directly to these operations, > > they should be inlined. > > > > > > And then if there are places that don't inline, they should > > call the "instantiations" in interface word/long. > > > > > > And then, furthermore, we should consider an interface > > for "Integer.Divide/Multiple/Remainder"? > > A place to park the possibly large code? > > > > > > - Jay > > > > > > ---------------------------------------- > > > From: hosking at cs.purdue.edu > > > Date: Wed, 10 Mar 2010 21:01:54 -0500 > > > To: jay.krell at cornell.edu > > > CC: m3commit at elegosoft.com > > > Subject: Re: [M3commit] CVS Update: cm3 > > > > > > I am confused. The intention is that Long.LeftShift is always inlined wherever it is called. Is the code for it really that hairy? > > > > > > On 10 Mar 2010, at 20:30, Jay K wrote: > > > > > >> > > >> If there exists an inline expansion, and it is large, but there is a function whose job it is to hold the expansion, whether it is inline or not, and the backend was informed of this function's name, and whether or not it was currently producing it, the backend could generate the inline expansion for Long__LeftShift, but otherwise generate calls to it. > > >> > > >> > > >> That is, e.g. m3cg.left_shift(type := int64|word64) could chose to either call Long__LeftShift or generate the body of Long__LeftShift. Not based on some "is inlining profitable heuristic", but specifically if told it is generating the function Long__LeftShift or not. > > >> > > >> > > >> That is, there's no point in having a C function m3_left_shift64 or somesuch, and having Long__LeftShift call it. Instead a backend should generate the body of Long__LeftShift inline when told it is generating that function, vs. generating a call to that function when it is otherwise asked to do a 64bit leftshift if it decides that inlining it every time is too large. > > >> > > >> Granted at least two things: > > >> Given 64bit target, Long__LeftShift vs. Word__LeftShift is ambiguous. > > >> And I'm inlined to just always inline anyway. > > >> > > >> > > >> (I still don't like the term "Long" here. It doesn't convey unsigned.) > > >> > > >> > > >> - Jay > > >> > > >> > > >> > > >> ________________________________ > > >>> Subject: Re: [M3commit] CVS Update: cm3 > > >>> From: hosking at cs.purdue.edu > > >>> Date: Wed, 10 Mar 2010 19:57:17 -0500 > > >>> CC: m3commit at elegosoft.com > > >>> To: jay.krell at cornell.edu > > >>> > > >>> > > >>> > > >>> Yes, someone can pass the function as a parameter. > > >>> > > >>> I don't understand the rest of what you are saying. > > >>> > > >>> > > >>> Antony Hosking | Associate Professor | Computer Science | Purdue University > > >>> 305 N. University Street | West Lafayette | IN 47907 | USA > > >>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> On 10 Mar 2010, at 17:27, Jay K wrote: > > >>> > > >>> > > >>> Hey, should we maybe have a flag that indicates we are producing a function that is meant to provide and only provide exactly the functionality of the "one" m3cg call? > > >>> You know, that'd be a great hint to: > > >>> > > >>> - definitely definitely definitely definitely inline > > >>> - if we know this thing existed, we could use it instead of a .c file. > > >>> - along with the function's name for the invocations that aren't the function? > > >>> > > >>> > > >>> Why provide the function anyway? In case someone takes the address? > > >>> > > >>> > > >>> > > >>> - Jay > > >>> > > >>> > > >>> ________________________________ > > >>> From: jay.krell at cornell.edu > > >>> To: hosking at cs.purdue.edu > > >>> Date: Mon, 8 Mar 2010 16:04:45 +0000 > > >>> CC: m3commit at elegosoft.com > > >>> Subject: Re: [M3commit] CVS Update: cm3 > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> Fair enough. > > >>> > > >>> > > >>> > > >>> - Jay > > >>> > > >>> > > >>> > > >>> ________________________________ > > >>> > > >>> From: hosking at cs.purdue.edu > > >>> Date: Mon, 8 Mar 2010 10:29:00 -0500 > > >>> To: jay.krell at cornell.edu > > >>> CC: m3commit at elegosoft.com > > >>> Subject: Re: [M3commit] CVS Update: cm3 > > >>> > > >>> > > >>> > > >>> > > >>> You can't call the support function Long__Extract because we already have a Long__Extract in m3core/src/word. It is the library representative of the front-end compiler-generated (inlined) Long.Extract. > > >>> > > >>> > > >>> > > >>> On 8 Mar 2010, at 07:30, Jay K wrote: > > >>> > > >>> > > >>> > > >>> diff attached > > >>> > > >>> > > >>> If folks really want to use tables or function calls here, let me know. > > >>> > > >>> > > >>> The data was historically problematic, though we worked out the problems. > > >>> - It was initialized at runtime which has a race condition, fixed years ago. > > >>> - It can't be "exported" on NT386 (which is the only platform that uses it) and must be statically linked. > > >>> > > >>> The data is still used by Word.Insert and Long.Insert is still a function, but those > > >>> are my next targets. > > >>> > > >>> > > >>> This is a case where the user does write a function call. Word.Extract or Long.Extract. > > >>> (So the function should have been called Long__Extract.) > > >>> > > >>> > > >>> - Jay > > >>> > > >>> Date: Mon, 8 Mar 2010 13:26:36 +0000 > > >>> To: m3commit at elegosoft.com > > >>> From: jkrell at elego.de > > >>> Subject: [M3commit] CVS Update: cm3 > > >>> > > >>> CVSROOT: /usr/cvs > > >>> Changes by: jkrell at birch. 10/03/08 13:26:36 > > >>> > > >>> Modified files: > > >>> cm3/m3-sys/m3back/src/: M3x86.m3 Stackx86.i3 Stackx86.m3 > > >>> > > >>> Log message: > > >>> "rewrite" extract to not use tables and always be inlined for 64bit > > >>> > > >>> equivalent C code: > > >>> > > >>> UT __stdcall extract(UT x, uint32 offset, uint32 count) > > >>> { > > >>> x>>= offset; > > >>> x &= ~((~(UT)0) << count); > > >>> return x; > > >>> } > > >>> > > >>> extract32 > > >>> 00000729: 8B4DEC MOV ECX tv.126[_m] > > >>> 0000072C: 8B5DF4 MOV EBX tv.124[_a32] > > >>> 0000072F: D3EB SHR EBX > > >>> 00000731: 8BD1 MOV EDX ECX This is pointless and I'll try to remove. > > >>> 00000733: 8B4DE4 MOV ECX tv.128[_n] > > >>> 00000736: BEFFFFFFFF MOV ESI $4294967295 I'll look for smaller form of this. or esi -1? > > >>> 0000073B: D3E6 SHL ESI > > >>> 0000073D: F7D6 NOT ESI > > >>> 0000073F: 23DE AND EBX ESI > > >>> > > >>> extract64 > > >>> 000008E4: 8B4DD8 MOV ECX tv.134[_m] > > >>> 000008E7: 8B5DE8 MOV EBX tv.131[_a64] > > >>> 000008EA: 8B55EC MOV EDX tv.131[_a64]+4 > > >>> 000008ED: 0FADD3 SHRD EBX EDX ECX > > >>> 000008F0: D3EA SHR EDX > > >>> 000008F2: F6C120 TEST ECX $32 > > >>> 000008F5: 7400 JE rel8 L.107 > > >>> 000008F7: 8BDA MOV EBX EDX > > >>> 000008F9: 33D2 XOR EDX EDX > > >>> set_label L.107 > > >>> 000008FB: 8BF1 MOV ESI ECX This is pointless and I'll try to remove. > > >>> 000008FD: 8B4DD0 MOV ECX tv.136[_n] > > >>> 00000900: BFFFFFFFFF MOV EDI $4294967295 I'll look for smaller form of this. or edi -1? > > >>> 00000905: B8FFFFFFFF MOV EAX $4294967295 I'll look for smaller form of this. (heck, at least mov edi, eax) > > >>> 0000090A: 0FA5F8 SHLD EAX EDI ECX > > >>> 0000090D: D3E7 SHL EDI > > >>> 0000090F: F6C120 TEST ECX $32 > > >>> 00000912: 7400 JE rel8 L.108 > > >>> 00000914: 8BC7 MOV EAX EDI > > >>> 00000916: 33FF XOR EDI EDI > > >>> set_label L.108 > > >>> 00000918: F7D7 NOT EDI > > >>> 0000091A: F7D0 NOT EAX > > >>> 0000091C: 23DF AND EBX EDI > > >>> 0000091E: 23D0 AND EDX EAX > > >>> > > >>> having n or m and n (or just m? I think so.) be constant leads to better code > > >>> > > >>> some other small cleanup, like avoiding calling find twice, > > >>> I don't see why it was that way > > >>> > > >>> <1.txt> > > >>> > > >>> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Thu Mar 11 23:03:30 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 17:03:30 -0500 Subject: [M3devel] Target.EOL In-Reply-To: References: Message-ID: I don't think I've used Target.EOL, but I've definitely used some EOL definition somewhere, maybe it was Wr.EOL, I'd have to check to be sure. I'm not sure the implications of removing Target.EOL, but whatever is done, we need to maintain a way to distinguish at runtime the proper line ending format for the host environment. I've written a lot of code in the past where this is important. Some of my code interfaces with other systems whose line ending formats have to be compared and/or reconciled with the host when doing serial data transfer (yes there are still systems that rely on good 'ol serial I/O, particularly legacy DoD systems). Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 10:41 AM To: m3devel Subject: [M3devel] Target.EOL Importance: Low I strongly suspect that Target.EOL can go away and just use Wr.EOL instead. Or even just \n. Cross builds are fairly rare, esp. cross between NT and Posix, and very many tools treat \n, \r\n, and perhaps \r the same, so even crossing NT <=> Posix doesn't likely matter. ie. it works, assuming you have a cross m3cg (ie: mingwin/cygwin hosted for NT hosts) gcc, Visual C++ compiler, m3front, all treat \r and \r\n the same. (witness all our *.h, *.c, *.i3, *.m3 files use \r\n) I think we have some tools that don't understand \r\n, but in my opinion that is a bug. All three formats should be treated the same. I know notepad doesn't display \n well, and Sun cc might not like \r\n. I know some compiler I used recently was picky, but "many" compilers are not. cmd might be ok with \n. Python calls it "universal newlines", treating all three formats the same. C:\dev2\cm3.2\m3-sys\cm3\src\Utils.m3(190):PROCEDURE CopyText (old, new: TEXT) = C:\dev2\cm3.2\m3-sys\m3middle\src\M3File.m3(61):PROCEDURE CopyText (src, dest: TEXT; eol: TEXT) RAISES {OSError.E} = probably leave alone, except that they don't treat \r correctly if you consider the historical Apple behavior. There are three formats, not two. Anyway, it's not a big deal. I think my problem here is solved by the change to m3x86.m3 I already made. I'll put Target.EOL back on my machine. - Jay ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 12 01:25:15 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 00:25:15 +0000 Subject: [M3devel] Target.EOL In-Reply-To: References: , Message-ID: Serial I/O is used plenty, for debugging. Though the protocols are generally "binary" not "text". Even any world with multiple systems messes with historical notions that one system is one way, other systems are another, and that they rarely see each other or exchange files. File exchange is very common these days, so it behooves every piece of new code to understand either format and possibly detect what the old code on the other side wants and be willing to write any of the three formats. "Constants" end up not useful. This isn't the EOL you care about. Target.EOL is written to .m3ship, foo.molog. It is used in m3front, m3middle, m3back, cm3 packages. It is exposed from m3middle/Target.i3. It is used by the old bootstrapping code, which is strewn around cm3, which I do use some of as part of my automation, uses it. (I don't use e.g. the makefile fragments that it outputs, but I use the behavior that stops after producing assembly, doesn't run the assembler or linker or C compiler) For example if you cross from NT386 to Posix, any \r\n in .man, .c, .h files will be changed to \n, and copied to the target directory. The thing is though, we don't have any \r\n anyway so that is a no-op. If you cross from Posix to NT386, \n will changed to \r\n. But it doesn't matter. Native builds just leave the \n alone -- which is what we always have -- and they work fine. If you cross from a hypothetical old Macintosh with its \r format, the newlines get completely destroyed because the code is wrong. Oops. Target.EOL can be changed to Wr.EOL or a hardcoded \n and almost nothing would change. The difference is that if we hardcoded \n, native NT386 users who happened to look at .m3ship with notepad, wouldn't have a good experience. Almost nobody ever looks at those files. Use Wr.EOL and then the only downside would be .m3ship files when crossing to NT386. And even still, .m3ship files don't participate in cross builds as I do them. My cross build only produces cm3, and then I build the rest native. Though there is a place for this perhaps. A cross build that builds everything, leaving final assembly, C compilation, linking and shipping to the target. We should consider that as a future distribution format. Similar to today's packages, but cross. But still, Wr.EOL will be fine. I'm very tempted to say the same thing about / and \, except that I recently experienced the pain of using an older NT386 cm3 that doesn't accept \. It behooves me to use \ "where appropriate" so I can be compatible with that. I need to fix scripts/python and/or cminstall/src/config-no-install a little. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Thu, 11 Mar 2010 17:03:30 -0500 Subject: RE: [M3devel] Target.EOL I don?t think I?ve used Target.EOL, but I?ve definitely used some EOL definition somewhere, maybe it was Wr.EOL, I?d have to check to be sure. I?m not sure the implications of removing Target.EOL, but whatever is done, we need to maintain a way to distinguish at runtime the proper line ending format for the host environment. I?ve written a lot of code in the past where this is important. Some of my code interfaces with other systems whose line ending formats have to be compared and/or reconciled with the host when doing serial data transfer (yes there are still systems that rely on good ?ol serial I/O, particularly legacy DoD systems). Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 10:41 AM To: m3devel Subject: [M3devel] Target.EOL Importance: Low I strongly suspect that Target.EOL can go away and just use Wr.EOL instead. Or even just \n. Cross builds are fairly rare, esp. cross between NT and Posix, and very many tools treat \n, \r\n, and perhaps \r the same, so even crossing NT <=> Posix doesn't likely matter. ie. it works, assuming you have a cross m3cg (ie: mingwin/cygwin hosted for NT hosts) gcc, Visual C++ compiler, m3front, all treat \r and \r\n the same. (witness all our *.h, *.c, *.i3, *.m3 files use \r\n) I think we have some tools that don't understand \r\n, but in my opinion that is a bug. All three formats should be treated the same. I know notepad doesn't display \n well, and Sun cc might not like \r\n. I know some compiler I used recently was picky, but "many" compilers are not. cmd might be ok with \n. Python calls it "universal newlines", treating all three formats the same. C:\dev2\cm3.2\m3-sys\cm3\src\Utils.m3(190):PROCEDURE CopyText (old, new: TEXT) = C:\dev2\cm3.2\m3-sys\m3middle\src\M3File.m3(61):PROCEDURE CopyText (src, dest: TEXT; eol: TEXT) RAISES {OSError.E} = probably leave alone, except that they don't treat \r correctly if you consider the historical Apple behavior. There are three formats, not two. Anyway, it's not a big deal. I think my problem here is solved by the change to m3x86.m3 I already made. I'll put Target.EOL back on my machine. - Jay CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Highjinks at gmx.com Fri Mar 12 01:35:25 2010 From: Highjinks at gmx.com (Chris) Date: Fri, 12 Mar 2010 01:35:25 +0100 (CET) Subject: [M3devel] Constant to Constant? Message-ID: <20100311203930.9b7cb406.Highjinks@gmx.com> Things are progressing here, slowly but steadily. I'm curious... When writing an interface for something like this ... /* C Type */ const C_Foo_t *Bar = get_foo_data(misc); Is it better to do it this way... <* EXTERNAL get_foo_data:C *> TYPE Foo_Data = PROCEDURE(get_foo_data(somemisc : misctype) : C_Foo_t; Or should I just do... UNSAFE INTERFACE Foo; (* Translating the C typedef struct over to Modula3 code. *) TYPE C_Foo_T = UNTRACED REF RECORD .... END; <* EXTERNAL get_foo_data:C *> PROCEDURE get_foo_data(somemisc : misctype) : C_foo_t; END Foo. Main.m3 IMPORT Foo; Foo_Data := Foo.get_foo_data(Misc); END Main. C_Foo_t is known, and has a Modula 3 representation in the Interface. But as the C Code shows, it has to be a constant value. What's the best way to do this without hosing the Interface for everyone else that might want to use it? Variable expressions are no problem, it's making it a constant that's giving me trouble. It's constant because the data structure is managed by the supporting library, not by the Modula3 Code. Tips...pointers? -- Chris From jay.krell at cornell.edu Fri Mar 12 02:06:57 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 01:06:57 +0000 Subject: [M3devel] Constant to Constant? In-Reply-To: <20100311203930.9b7cb406.Highjinks@gmx.com> References: <20100311203930.9b7cb406.Highjinks@gmx.com> Message-ID: Chris, I'm not sure I understand. Is the result "in misc", or is it constant relative to the program's lifetime? Or is it initialized on-demand and then constant from then on? There are a few instances where "constants" are initialized in Modula-3 module initializers. They are "VAR" from a technical point of view, in the .i3 file, but then VAR is immediately preceded by a comment that they are CONST. I do something similar in m3core/unix, except the data is actually const/static initialized in C code. I do that to avoid duplicating header content. Header content can be duplicated, if it is fairly portable and stable, only needs to be duplicated once, easy to get correct, and stay correct. The particular problem in m3core/unix is portability -- the header content would have to be duplicated differently for every target. Incorrectly duplicated C headers silently violate safety (see below). You should try to provide a SAFE interface, so that client code can be SAFE. And then, you are responsible for upholding what that means. Your module implementation might be UNSAFE. But clients must be protected from a "certain class of bug". For example, a claimed-to-be SAFE interface cannot expose stdlib/free() because a client could double free. This is an easy thing to mess up and it is very unfortunate when it is. It is roughly equivalent to bugs in the compiler or garbage collector. - Jay > From: Highjinks at gmx.com > To: m3devel at elegosoft.com > Date: Fri, 12 Mar 2010 01:35:25 +0100 > Subject: [M3devel] Constant to Constant? > > > Things are progressing here, slowly but steadily. > > I'm curious... > > When writing an interface for something like this ... > > /* C Type */ > const C_Foo_t *Bar = get_foo_data(misc); > > Is it better to do it this way... > > <* EXTERNAL get_foo_data:C *> > TYPE Foo_Data = PROCEDURE(get_foo_data(somemisc : misctype) : C_Foo_t; > > Or should I just do... > UNSAFE INTERFACE Foo; > > (* Translating the C typedef struct over to Modula3 code. *) > TYPE C_Foo_T = UNTRACED REF RECORD .... END; > > <* EXTERNAL get_foo_data:C *> > PROCEDURE get_foo_data(somemisc : misctype) : C_foo_t; > END Foo. > > Main.m3 > IMPORT Foo; > Foo_Data := Foo.get_foo_data(Misc); > END Main. > > C_Foo_t is known, and has a Modula 3 representation in the Interface. But as the C Code shows, it has to be a constant value. > > What's the best way to do this without hosing the Interface for everyone else that might want to use it? Variable expressions are no problem, it's making it a constant that's giving me trouble. It's constant because the data structure is managed by the supporting library, not by the Modula3 Code. > > Tips...pointers? > > -- > Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Mar 12 02:14:12 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 20:14:12 -0500 Subject: [M3devel] Ticket #1082 Message-ID: I have updated the comments on Ticket #1082 There are two postings, as shown below. Note that it looks like there is a problem with "START /WAIT ..." command differences between Windows versions, so we may need to open up a new Trac ticket for this issue (see comment #2 below). FIRST COMMENT: --------------------- The "startReactor.CMD" file is obsolete and no longer in the repository. The new files are in "scripts\install\windows" and should be copied to the "cm3\bin" folder on a windows installation. "cm3StartIDE.CMD" replaces "startReactor.CMD". This file makes use of "cm3CommandShell.CMD", a script that ensures the environment is set up properly, including Visual C++. It appears that the problem reporter has his installation rooted at E:\cm3 rather than C:\cm3. So, he will need to set an environment variable "set CM3_ROOT=E:\cm3" to let these scripts know where to find the cm3 installation. Alternately, cm3CommandShell can be invoked with the arguments "Root E:\cm3" to tell it where to find the installation. Today, I checked in an update to cm3CommandShell.CMD that allows it to search the current PATH environment variable in an attempt to find the root of the cm3 installation, so now if "E:\cm3\bin" is in your path, it will find it without having to set the CM3_ROOT or invoking with "Root E:\cm3". Hope this helps! Note also that strictly speaking, these CMD files are not necessary; rather they are conveniences I've found useful under Windows. I also can supply some .REG files that will allow you to integrate these into the Explorer shell so that you can start a cm3 command shell in any folder and you can start CM3IDE. SECOND COMMENT: --------------------- Ok as for CM3IDE timing out, I've been able to reproduce this problem on Vista. There seems to be some difference I don't understand yet between the way the various Windows versions deal with the "START /WAIT ..." command. Here is a "quick fix" until we come up with a better solution. Note that the downside here is that when you terminate your browser, CM3IDE will stay running, instead of terminating. You can restart your browser and point it back to the http://localhost:3800/ URL and verify that it reconnects to CM3IDE. So, to terminate CM3IDE, you will have to use CTRL-C from its console output window. Now for the quick fix: 1. Go to your CM3_IDE_HOME folder (e.g., C:\Users\RColeburn\Documents\CM3_IDE_Home). 2. Open the most recent CM3_IDE.cfg file in notepad (e.g., notepad CM3_IDE.cfg1). 3. Edit the line toward the end of the file that begins with "proc start_browser". On this line change TRUE to FALSE. Note that case is significant. Now try cm3IDE.exe, or cm3_StartIDE.CMD. It should work for you now as described above. Regards, Randy Coleburn ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Mar 12 02:38:34 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 20:38:34 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait Message-ID: Ok, I've done a bit more testing. I don't think the problem has to do with "START /WAIT ..." but rather it seems to be a difference in Windows Internet Explorer. When your browser is set to FIREFOX instead of IE, the "START /WAIT firefox.exe" command works fine, so you can keep the "return TRUE" in the CM3_IDE.cfg file. But, when using IE8, the "START /WAIT iexplorer.exe" command does not wait for IE8 to terminate and returns immediately, thus the "return TRUE" in CM3_IDE.cfg must be changed to "return FALSE" to prevent immediate shutdown of CM3IDE. If I go to a Windows 2000 box running IE6, the "START /WAIT" again seems to wait for IE to terminate. I haven't tested yet with IE7 to see if it behaves correctly or not. So for now, folks may want to use FireFox instead of IE with CM3IDE. Regards, Randy ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 12 02:43:03 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 01:43:03 +0000 Subject: [M3devel] more info on ticket 1082 wrt start /wait In-Reply-To: References: Message-ID: What if you just simply: "start http://localhost:whateverport" Personally I find cm3ide/reactor to be ..not great. The behavior you are noticing is that the web browser might decide there is another adequate instance of it running and ask it to display something and exit. You will likely find different behavior depending on if an instance of the browser is already running. There might also be a command line option to control this. But I think your best best is just not to use /wait. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Thu, 11 Mar 2010 20:38:34 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait Ok, I?ve done a bit more testing. I don?t think the problem has to do with ?START /WAIT ?? but rather it seems to be a difference in Windows Internet Explorer. When your browser is set to FIREFOX instead of IE, the ?START /WAIT firefox.exe? command works fine, so you can keep the ?return TRUE? in the CM3_IDE.cfg file. But, when using IE8, the ?START /WAIT iexplorer.exe? command does not wait for IE8 to terminate and returns immediately, thus the ?return TRUE? in CM3_IDE.cfg must be changed to ?return FALSE? to prevent immediate shutdown of CM3IDE. If I go to a Windows 2000 box running IE6, the ?START /WAIT? again seems to wait for IE to terminate. I haven?t tested yet with IE7 to see if it behaves correctly or not. So for now, folks may want to use FireFox instead of IE with CM3IDE. Regards, Randy CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 12 02:36:53 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 01:36:53 +0000 Subject: [M3devel] Ticket #1082 In-Reply-To: References: Message-ID: > There seems to be some difference I don't understand yet between > the way the various Windows versions deal with the "START /WAIT ..." command. Please elaborate. What varying behaviors are you seeing on different versions? I have used start /wait a fair number of times and never noticed any differences among Windows versions, at least on NT versions since circa NT 4. Definitely maybe some versions don't have any sort of start nor /wait, and start on Win9x is probably different than NT. start is a builtin command to NT cmd but an .exe on Win9x, as I vaguely recall. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Thu, 11 Mar 2010 20:14:12 -0500 Subject: [M3devel] Ticket #1082 I have updated the comments on Ticket #1082 There are two postings, as shown below. Note that it looks like there is a problem with ?START /WAIT ?? command differences between Windows versions, so we may need to open up a new Trac ticket for this issue (see comment #2 below). FIRST COMMENT: --------------------- The "startReactor.CMD" file is obsolete and no longer in the repository. The new files are in "scripts\install\windows" and should be copied to the "cm3\bin" folder on a windows installation. "cm3StartIDE.CMD" replaces "startReactor.CMD". This file makes use of "cm3CommandShell.CMD", a script that ensures the environment is set up properly, including Visual C++. It appears that the problem reporter has his installation rooted at E:\cm3 rather than C:\cm3. So, he will need to set an environment variable "set CM3_ROOT=E:\cm3" to let these scripts know where to find the cm3 installation. Alternately, cm3CommandShell can be invoked with the arguments "Root E:\cm3" to tell it where to find the installation. Today, I checked in an update to cm3CommandShell.CMD that allows it to search the current PATH environment variable in an attempt to find the root of the cm3 installation, so now if "E:\cm3\bin" is in your path, it will find it without having to set the CM3_ROOT or invoking with "Root E:\cm3". Hope this helps! Note also that strictly speaking, these CMD files are not necessary; rather they are conveniences I?ve found useful under Windows. I also can supply some .REG files that will allow you to integrate these into the Explorer shell so that you can start a cm3 command shell in any folder and you can start CM3IDE. SECOND COMMENT: --------------------- Ok as for CM3IDE timing out, I've been able to reproduce this problem on Vista. There seems to be some difference I don't understand yet between the way the various Windows versions deal with the "START /WAIT ..." command. Here is a "quick fix" until we come up with a better solution. Note that the downside here is that when you terminate your browser, CM3IDE will stay running, instead of terminating. You can restart your browser and point it back to the http://localhost:3800/ URL and verify that it reconnects to CM3IDE. So, to terminate CM3IDE, you will have to use CTRL-C from its console output window. Now for the quick fix: 1. Go to your CM3_IDE_HOME folder (e.g., C:\Users\RColeburn\Documents\CM3_IDE_Home). 2. Open the most recent CM3_IDE.cfg file in notepad (e.g., notepad CM3_IDE.cfg1). 3. Edit the line toward the end of the file that begins with "proc start_browser". On this line change TRUE to FALSE. Note that case is significant. Now try cm3IDE.exe, or cm3_StartIDE.CMD. It should work for you now as described above. Regards, Randy Coleburn CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Mar 12 02:46:13 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 20:46:13 -0500 Subject: [M3devel] Constant to Constant? In-Reply-To: <20100311203930.9b7cb406.Highjinks@gmx.com> References: <20100311203930.9b7cb406.Highjinks@gmx.com> Message-ID: Chris: You may also want to look at pkg\m3core\src\win32\WinUser.i3 Regards, Randy -----Original Message----- From: Chris [mailto:Highjinks at gmx.com] Sent: Thursday, March 11, 2010 7:35 PM To: m3devel at elegosoft.com Subject: [M3devel] Constant to Constant? Things are progressing here, slowly but steadily. I'm curious... When writing an interface for something like this ... /* C Type */ const C_Foo_t *Bar = get_foo_data(misc); Is it better to do it this way... <* EXTERNAL get_foo_data:C *> TYPE Foo_Data = PROCEDURE(get_foo_data(somemisc : misctype) : C_Foo_t; Or should I just do... UNSAFE INTERFACE Foo; (* Translating the C typedef struct over to Modula3 code. *) TYPE C_Foo_T = UNTRACED REF RECORD .... END; <* EXTERNAL get_foo_data:C *> PROCEDURE get_foo_data(somemisc : misctype) : C_foo_t; END Foo. Main.m3 IMPORT Foo; Foo_Data := Foo.get_foo_data(Misc); END Main. C_Foo_t is known, and has a Modula 3 representation in the Interface. But as the C Code shows, it has to be a constant value. What's the best way to do this without hosing the Interface for everyone else that might want to use it? Variable expressions are no problem, it's making it a constant that's giving me trouble. It's constant because the data structure is managed by the supporting library, not by the Modula3 Code. Tips...pointers? -- Chris CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. From rcolebur at SCIRES.COM Fri Mar 12 03:06:36 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 21:06:36 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait In-Reply-To: References: Message-ID: Jay et al: Well I've done more testing. Jay you are right about the multiple instances. For both IE8 and FireFox on Vista, if you already have an instance running and use "START /WAIT" to get another instance, the second instance starts up (a new window) and the "START /WAIT" immediately returns rather than waiting. Conversely, if this is the first instance of IE8 or FireFox, then "START /WAIT" does indeed wait for the browser to close before it returns. I'll have to get to an XP machine to confirm if this same behavior occurs there as well. If you go back to IE6, it didn't have Tabbed windows, so maybe that is why "START /WAIT" always waited. I don't have an immediate fix for this behavior since the original design used the "START /WAIT" functionality on Windows. On POSIX, it doesn't. The quick fix I mentioned earlier is probably the best thing to do now. I could perhaps modify the sources to change the default CFG to "return FALSE" on Windows instead of "return TRUE", thereby implementing the quick fix for everyone. I'll check into this shortly. As for your comments about cm3ide not being "great" that is your opinion; however, back in the day when reactor (aka cm3ide) first came out, this was indeed leading edge stuff. The idea that you could tweak your IDE by making simple HTML file changes I think was pretty novel at the time, and the thinking from the CMass Inc folks was that as the browser improved, your IDE would improve automatically (e.g. multiple tabs was a subsequent browser improvement). Even though you may not consider CM3IDE to be a great IDE, I do find that it is extremely helpful during development when browsing source code and looking up interface specs. With just a hyperlink, you can quickly find what you are looking for. It also has a lot of built-in links to reference material and coding examples. Note that cm3ide hasn't really changed much at all since reactor first debuted. Now that we finally have the open-source release, we can modify it to make it better. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 8:43 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait What if you just simply: "start http://localhost:whateverport" Personally I find cm3ide/reactor to be ..not great. The behavior you are noticing is that the web browser might decide there is another adequate instance of it running and ask it to display something and exit. You will likely find different behavior depending on if an instance of the browser is already running. There might also be a command line option to control this. But I think your best best is just not to use /wait. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Thu, 11 Mar 2010 20:38:34 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait Ok, I've done a bit more testing. I don't think the problem has to do with "START /WAIT ..." but rather it seems to be a difference in Windows Internet Explorer. When your browser is set to FIREFOX instead of IE, the "START /WAIT firefox.exe" command works fine, so you can keep the "return TRUE" in the CM3_IDE.cfg file. But, when using IE8, the "START /WAIT iexplorer.exe" command does not wait for IE8 to terminate and returns immediately, thus the "return TRUE" in CM3_IDE.cfg must be changed to "return FALSE" to prevent immediate shutdown of CM3IDE. If I go to a Windows 2000 box running IE6, the "START /WAIT" again seems to wait for IE to terminate. I haven't tested yet with IE7 to see if it behaves correctly or not. So for now, folks may want to use FireFox instead of IE with CM3IDE. Regards, Randy ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 12 03:47:53 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 11 Mar 2010 21:47:53 -0500 Subject: [M3devel] Target.EOL In-Reply-To: References: Message-ID: <5F2763E1-801E-4EB2-BFAC-884EB53DC3C6@cs.purdue.edu> On 11 Mar 2010, at 17:03, Coleburn, Randy wrote: > I don?t think I?ve used Target.EOL, but I?ve definitely used some EOL definition somewhere, maybe it was Wr.EOL, I?d have to check to be sure. This is part of the compiler subsystem and its tool interfaces. I don't think it will affect application code. If we are to default I suggest we go with the more compact \n termination instead of verbose \r\n. > > I?m not sure the implications of removing Target.EOL, but whatever is done, we need to maintain a way to distinguish at runtime the proper line ending format for the host environment. > > I?ve written a lot of code in the past where this is important. Some of my code interfaces with other systems whose line ending formats have to be compared and/or reconciled with the host when doing serial data transfer (yes there are still systems that rely on good ?ol serial I/O, particularly legacy DoD systems). > > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Thursday, March 11, 2010 10:41 AM > To: m3devel > Subject: [M3devel] Target.EOL > Importance: Low > > I strongly suspect that Target.EOL can go away and just use Wr.EOL instead. > Or even just \n. > > > Cross builds are fairly rare, esp. cross between NT and Posix, and very > many tools treat \n, \r\n, and perhaps \r the same, so even > crossing NT <=> Posix doesn't likely matter. > ie. it works, assuming you have a cross m3cg (ie: mingwin/cygwin hosted for NT hosts) > > > gcc, Visual C++ compiler, m3front, all treat \r and \r\n the same. > (witness all our *.h, *.c, *.i3, *.m3 files use \r\n) > I think we have some tools that don't understand \r\n, but in my opinion that is a bug. > All three formats should be treated the same. > > > I know notepad doesn't display \n well, and Sun cc might not like \r\n. > I know some compiler I used recently was picky, but "many" compilers are not. > cmd might be ok with \n. > Python calls it "universal newlines", treating all three formats the same. > > > C:\dev2\cm3.2\m3-sys\cm3\src\Utils.m3(190):PROCEDURE CopyText (old, new: TEXT) = > C:\dev2\cm3.2\m3-sys\m3middle\src\M3File.m3(61):PROCEDURE CopyText (src, dest: TEXT; eol: TEXT) RAISES {OSError.E} = > > > probably leave alone, except that they don't treat \r correctly if you consider the historical Apple behavior. > There are three formats, not two. > > > Anyway, it's not a big deal. I think my problem here is solved by the change to m3x86.m3 I already made. > I'll put Target.EOL back on my machine. > > > - Jay > > CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. > > EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 12 03:49:39 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 02:49:39 +0000 Subject: [M3devel] more info on ticket 1082 wrt start /wait In-Reply-To: References: , , Message-ID: Start /wait is always doing the same thing, it is always waiting. But what it is launching is deciding to exit right away, or not. More later.. From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Thu, 11 Mar 2010 21:06:36 -0500 Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait Jay et al: Well I?ve done more testing. Jay you are right about the multiple instances. For both IE8 and FireFox on Vista, if you already have an instance running and use ?START /WAIT? to get another instance, the second instance starts up (a new window) and the ?START /WAIT? immediately returns rather than waiting. Conversely, if this is the first instance of IE8 or FireFox, then ?START /WAIT? does indeed wait for the browser to close before it returns. I?ll have to get to an XP machine to confirm if this same behavior occurs there as well. If you go back to IE6, it didn?t have Tabbed windows, so maybe that is why ?START /WAIT? always waited. I don?t have an immediate fix for this behavior since the original design used the ?START /WAIT? functionality on Windows. On POSIX, it doesn?t. The quick fix I mentioned earlier is probably the best thing to do now. I could perhaps modify the sources to change the default CFG to ?return FALSE? on Windows instead of ?return TRUE?, thereby implementing the quick fix for everyone. I?ll check into this shortly. As for your comments about cm3ide not being ?great? that is your opinion; however, back in the day when reactor (aka cm3ide) first came out, this was indeed leading edge stuff. The idea that you could tweak your IDE by making simple HTML file changes I think was pretty novel at the time, and the thinking from the CMass Inc folks was that as the browser improved, your IDE would improve automatically (e.g. multiple tabs was a subsequent browser improvement). Even though you may not consider CM3IDE to be a great IDE, I do find that it is extremely helpful during development when browsing source code and looking up interface specs. With just a hyperlink, you can quickly find what you are looking for. It also has a lot of built-in links to reference material and coding examples. Note that cm3ide hasn?t really changed much at all since reactor first debuted. Now that we finally have the open-source release, we can modify it to make it better. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 8:43 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait What if you just simply: "start http://localhost:whateverport" Personally I find cm3ide/reactor to be ..not great. The behavior you are noticing is that the web browser might decide there is another adequate instance of it running and ask it to display something and exit. You will likely find different behavior depending on if an instance of the browser is already running. There might also be a command line option to control this. But I think your best best is just not to use /wait. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Thu, 11 Mar 2010 20:38:34 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait Ok, I?ve done a bit more testing. I don?t think the problem has to do with ?START /WAIT ?? but rather it seems to be a difference in Windows Internet Explorer. When your browser is set to FIREFOX instead of IE, the ?START /WAIT firefox.exe? command works fine, so you can keep the ?return TRUE? in the CM3_IDE.cfg file. But, when using IE8, the ?START /WAIT iexplorer.exe? command does not wait for IE8 to terminate and returns immediately, thus the ?return TRUE? in CM3_IDE.cfg must be changed to ?return FALSE? to prevent immediate shutdown of CM3IDE. If I go to a Windows 2000 box running IE6, the ?START /WAIT? again seems to wait for IE to terminate. I haven?t tested yet with IE7 to see if it behaves correctly or not. So for now, folks may want to use FireFox instead of IE with CM3IDE. Regards, Randy CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Mar 12 04:37:48 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 22:37:48 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait In-Reply-To: References: , , Message-ID: Jay: Yes, you said it better than I did. The problem is with the browser behavior, not with START /WAIT. I've implemented a quick-fix in the source code to change the default behavior on Windows to "return FALSE" on the "start_browser" function instead of "return TRUE". That way, regardless of whether or not the browser exits right away, CM3IDE will stay running until it is manually terminated via CTRL-C in its console output window. (Note: "return TRUE" means that CM3IDE will terminate when the browser terminates, whereas "return FALSE" means that CM3IDE will stay running even after the browser terminates. On a server system where you have multiple network clients connecting to the single CM3IDE instance running on the server, you would always want to use "return FALSE". Likewise, on a Unix system where you start CM3IDE as a background process when the system boots, you would want to use "return FALSE".) I am already thinking of alternate solutions and an easy way for CM3IDE to limit itself to a single instance. I'll work on polishing these up in the next few days, but the quick-fix should solve the immediate problem for now. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 9:50 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait Start /wait is always doing the same thing, it is always waiting. But what it is launching is deciding to exit right away, or not. More later.. ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Thu, 11 Mar 2010 21:06:36 -0500 Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait Jay et al: Well I've done more testing. Jay you are right about the multiple instances. For both IE8 and FireFox on Vista, if you already have an instance running and use "START /WAIT" to get another instance, the second instance starts up (a new window) and the "START /WAIT" immediately returns rather than waiting. Conversely, if this is the first instance of IE8 or FireFox, then "START /WAIT" does indeed wait for the browser to close before it returns. I'll have to get to an XP machine to confirm if this same behavior occurs there as well. If you go back to IE6, it didn't have Tabbed windows, so maybe that is why "START /WAIT" always waited. I don't have an immediate fix for this behavior since the original design used the "START /WAIT" functionality on Windows. On POSIX, it doesn't. The quick fix I mentioned earlier is probably the best thing to do now. I could perhaps modify the sources to change the default CFG to "return FALSE" on Windows instead of "return TRUE", thereby implementing the quick fix for everyone. I'll check into this shortly. As for your comments about cm3ide not being "great" that is your opinion; however, back in the day when reactor (aka cm3ide) first came out, this was indeed leading edge stuff. The idea that you could tweak your IDE by making simple HTML file changes I think was pretty novel at the time, and the thinking from the CMass Inc folks was that as the browser improved, your IDE would improve automatically (e.g. multiple tabs was a subsequent browser improvement). Even though you may not consider CM3IDE to be a great IDE, I do find that it is extremely helpful during development when browsing source code and looking up interface specs. With just a hyperlink, you can quickly find what you are looking for. It also has a lot of built-in links to reference material and coding examples. Note that cm3ide hasn't really changed much at all since reactor first debuted. Now that we finally have the open-source release, we can modify it to make it better. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 8:43 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait What if you just simply: "start http://localhost:whateverport" Personally I find cm3ide/reactor to be ..not great. The behavior you are noticing is that the web browser might decide there is another adequate instance of it running and ask it to display something and exit. You will likely find different behavior depending on if an instance of the browser is already running. There might also be a command line option to control this. But I think your best best is just not to use /wait. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Thu, 11 Mar 2010 20:38:34 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait Ok, I've done a bit more testing. I don't think the problem has to do with "START /WAIT ..." but rather it seems to be a difference in Windows Internet Explorer. When your browser is set to FIREFOX instead of IE, the "START /WAIT firefox.exe" command works fine, so you can keep the "return TRUE" in the CM3_IDE.cfg file. But, when using IE8, the "START /WAIT iexplorer.exe" command does not wait for IE8 to terminate and returns immediately, thus the "return TRUE" in CM3_IDE.cfg must be changed to "return FALSE" to prevent immediate shutdown of CM3IDE. If I go to a Windows 2000 box running IE6, the "START /WAIT" again seems to wait for IE to terminate. I haven't tested yet with IE7 to see if it behaves correctly or not. So for now, folks may want to use FireFox instead of IE with CM3IDE. Regards, Randy ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 12 07:45:07 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 06:45:07 +0000 Subject: [M3devel] merits/demerits of cm3ide In-Reply-To: References: , , Message-ID: > cm3ide > more later Being at the mercy of the rapidly changing web browsers is a very sharp double edged sword. An IDE with no editor and no debugger..is not an IDE. Maybe it is a source browser. Maybe an IDE is not worth it. People are very slow to change editors. Debuggers are hard to write. Probably better to try integrating with an actual IDE like VisualStudio or Eclipse. Or to have a decent enough language, library, build system, that the language doesn't need such costly support. Maybe we do have such a think. Hyperlinking the source is nice, but all I need is "find in files". I can open the documentation from a file system browser, which open it in a web browser. I should just be able to use online documentation anyway, not local stuff. Having the real compiler output enough information such that writing other lanugage-aware tools, such as hyperlinked source generation, is also good. Too many systems have their IDE write another parser, and no two parsers agree. However it is a tough problem, because the compiler's job is different than a source browser. For example, a compiler must reject incorrect source, but a source browser should be lenient. Relying on the browser is a cheap way to get a somewhat decent somewhat portable very limited gui. That's about it. I don't think it is fixable. It's just pretty much all wierd and wrong. I was quite shocked the first time I used it, having heard it described as an IDE. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Mar 12 10:20:59 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Fri, 12 Mar 2010 04:20:59 -0500 Subject: [M3devel] merits/demerits of cm3ide In-Reply-To: References: , , Message-ID: Jay as far as editor goes, CM3IDE lets you use the editor of your choice. Once you define it, CM3IDE calls upon your defined editor when you want to edit a source file. Note also that when you compile and get errors, CM3IDE annotates the listing and gives hyperlinks to allow you to make edits. When you click on a line with a syntax error and choose to edit it, CM3IDE calls upon your editor and tells it to position the cursor at that line so you can make the fix easily. CM3IDE also shows for each interface all modules that import that interface. This is useful also when you are making an interface change that will break modules, you can quickly find all the ones affected and make the edits. Granted you can use command line tools for finding stuff in files, but I personally like the hyper linking. In my environment, I use CM3IDE, the CRiSP programmers editor, and TortoiseCVS. CRiSP integrates with CVS so you can check files in/out of the repository automatically, all controlled from CM3IDE. There is no point in arguing further about merits/demerits of CM3IDE. I didn't write it; I simply worked to make it open-source. If you don't like it, don't use it. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, March 12, 2010 1:45 AM To: Coleburn, Randy; m3devel Subject: merits/demerits of cm3ide > cm3ide > more later Being at the mercy of the rapidly changing web browsers is a very sharp double edged sword. An IDE with no editor and no debugger..is not an IDE. Maybe it is a source browser. Maybe an IDE is not worth it. People are very slow to change editors. Debuggers are hard to write. Probably better to try integrating with an actual IDE like VisualStudio or Eclipse. Or to have a decent enough language, library, build system, that the language doesn't need such costly support. Maybe we do have such a think. Hyperlinking the source is nice, but all I need is "find in files". I can open the documentation from a file system browser, which open it in a web browser. I should just be able to use online documentation anyway, not local stuff. Having the real compiler output enough information such that writing other lanugage-aware tools, such as hyperlinked source generation, is also good. Too many systems have their IDE write another parser, and no two parsers agree. However it is a tough problem, because the compiler's job is different than a source browser. For example, a compiler must reject incorrect source, but a source browser should be lenient. Relying on the browser is a cheap way to get a somewhat decent somewhat portable very limited gui. That's about it. I don't think it is fixable. It's just pretty much all wierd and wrong. I was quite shocked the first time I used it, having heard it described as an IDE. - Jay ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Fri Mar 12 15:17:34 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 12 Mar 2010 15:17:34 +0100 Subject: [M3devel] merits/demerits of cm3ide In-Reply-To: References: , , Message-ID: <20100312151734.r2i0ui5nkg0k0woc@mail.elegosoft.com> Well, some people have used Reactor/cm3ide and liked it. There have been several requests for it in recent years. It's certainly no match for a large IDE framework like Eclipse, but it's the only GUI for the compiler we have. As for it not being an IDE, I do not really agree; it should already integrate compiling, shipping, source viewing, searching, pretty printing. And it integrates on the level of language elements likes types, interfaces etc. We should also be able to integrate a debugger (at least m3gdb); this has been done to gdb with other GUIs, too. I'd also like to see an M3 plugin for Eclipse, but nobody seems to be working on that. I myself am still happy with my XEmacs editor whenever I actually get to browse or write code ;-) So just view it as an offer and keep not using it if you don't like it. There are so many things that need work in the cm3 system that work won't get short during the next years. Olaf Quoting Jay K : > > cm3ide > > more later > > Being at the mercy of the rapidly changing web browsers is a very > sharp double edged sword. > > An IDE with no editor and no debugger..is not an IDE. > Maybe it is a source browser. > > Maybe an IDE is not worth it. > > People are very slow to change editors. > > Debuggers are hard to write. > > Probably better to try integrating with an actual IDE like > VisualStudio or Eclipse. > > Or to have a decent enough language, library, build system, that > the language doesn't > need such costly support. Maybe we do have such a think. > > Hyperlinking the source is nice, but all I need is "find in files". > I can open the documentation from a file system browser, which > open it in a web browser. I should just be able to use online > documentation anyway, not local stuff. > > Having the real compiler output enough information > such that writing other lanugage-aware tools, such > as hyperlinked source generation, is also good. > Too many systems have their IDE write another parser, > and no two parsers agree. However it is a tough problem, > because the compiler's job is different than a source > browser. For example, a compiler must reject incorrect > source, but a source browser should be lenient. > > Relying on the browser is a cheap way to get a somewhat > decent somewhat portable very limited gui. > That's about it. > > I don't think it is fixable. > It's just pretty much all wierd and wrong. > > I was quite shocked the first time I used it, having > heard it described as an IDE. -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From rcolebur at SCIRES.COM Sat Mar 13 06:57:00 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sat, 13 Mar 2010 00:57:00 -0500 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20100313011125.34CF12474001@birch.elegosoft.com> Message-ID: Jay: This is example code. The program terminates after the message box is displayed. There is no further processing. I did not write this example; I merely fixed it to compile again after someone changed the names of the procedures in the M3toC interface. Nevertheless, it is probably a bad "example" to foist on people by not also following the established pattern of freeing the storage , so I'll make a modification for this shortly. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, March 12, 2010 11:47 PM To: m3commit at elegosoft.com; Coleburn, Randy Subject: RE: [M3commit] CVS Update: cm3 This change is suspicious. I think it leaks. Please see http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/os/WIN32/FSWin32.m3?rev=1.7;content-type=text%2Fplain for the pattern that is usually used. Granted, given the conservativeness of the collector -- it finds stuff in registers and the stack without a need for metadata telling it where the points are, and the fact that we no longer have VM-synchronization, I don't understand the need for the usual pattern. Maybe because it is willing to move stuff? Therefore a non-traced copy is made for C that won't ever move? - Jay > Date: Sat, 13 Mar 2010 02:11:24 +0000 > To: m3commit at elegosoft.com > From: rcoleburn at elego.de > Subject: [M3commit] CVS Update: cm3 > > CVSROOT: /usr/cvs > Changes by: rcoleburn at birch. 10/03/13 02:11:24 > > Modified files: > cm3/examples/calling-c-win32/src/: OK.m3 > > Log message: > M3toC.TtoS has been renamed to M3toC.SharedTtoS, so make adjustments to Ok.m3 so that it will compile.--R.Coleburn > ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 13 07:36:25 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 13 Mar 2010 06:36:25 +0000 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20100313011125.34CF12474001@birch.elegosoft.com>, , Message-ID: Bad examples beget bad code. Bad (and good) examples get copied into real code. Don't make them. It wasn't a rename, it was I believe a semantic change accompanied by a rename. Better to not compile than compile and be incorrect. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Sat, 13 Mar 2010 00:57:00 -0500 Subject: Re: [M3devel] [M3commit] CVS Update: cm3 Jay: This is example code. The program terminates after the message box is displayed. There is no further processing. I did not write this example; I merely fixed it to compile again after someone changed the names of the procedures in the M3toC interface. Nevertheless, it is probably a bad ?example? to foist on people by not also following the established pattern of freeing the storage , so I?ll make a modification for this shortly. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, March 12, 2010 11:47 PM To: m3commit at elegosoft.com; Coleburn, Randy Subject: RE: [M3commit] CVS Update: cm3 This change is suspicious. I think it leaks. Please see http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/os/WIN32/FSWin32.m3?rev=1.7;content-type=text%2Fplain for the pattern that is usually used. Granted, given the conservativeness of the collector -- it finds stuff in registers and the stack without a need for metadata telling it where the points are, and the fact that we no longer have VM-synchronization, I don't understand the need for the usual pattern. Maybe because it is willing to move stuff? Therefore a non-traced copy is made for C that won't ever move? - Jay > Date: Sat, 13 Mar 2010 02:11:24 +0000 > To: m3commit at elegosoft.com > From: rcoleburn at elego.de > Subject: [M3commit] CVS Update: cm3 > > CVSROOT: /usr/cvs > Changes by: rcoleburn at birch. 10/03/13 02:11:24 > > Modified files: > cm3/examples/calling-c-win32/src/: OK.m3 > > Log message: > M3toC.TtoS has been renamed to M3toC.SharedTtoS, so make adjustments to Ok.m3 so that it will compile.--R.Coleburn > CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Sat Mar 13 07:49:55 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sat, 13 Mar 2010 01:49:55 -0500 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20100313011125.34CF12474001@birch.elegosoft.com>, , Message-ID: Jay: I agree "bad examples beget bad code", but I don't need to be lectured. I didn't create this example code. The example code is from the "Reactor v4.1 days and was produced by Critical Mass, Inc." Regardless of how the M3toC interface has changed since that time, it is obvious that the example code was not updated to be consistent with the changes. Thus, we had an example that failed to even compile, and that's not good either, especially since CM3IDE creates a copy of the example package in your private repository and lets you try to build and run it. I have repaired this example so that it compiles and runs. I've also made the change to free the memory to serve as a better exemplar of proper practice, even though it really doesn't affect the running of the program since the program terminates after the message box is closed and no further processing is done. I agree it was bad style in the first place on the part of the folks at Critical Mass. Now it is better. You will probably find other bad style examples, so if you want to fix them, be my guest. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Saturday, March 13, 2010 1:36 AM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] [M3commit] CVS Update: cm3 Bad examples beget bad code. Bad (and good) examples get copied into real code. Don't make them. It wasn't a rename, it was I believe a semantic change accompanied by a rename. Better to not compile than compile and be incorrect. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Sat, 13 Mar 2010 00:57:00 -0500 Subject: Re: [M3devel] [M3commit] CVS Update: cm3 Jay: This is example code. The program terminates after the message box is displayed. There is no further processing. I did not write this example; I merely fixed it to compile again after someone changed the names of the procedures in the M3toC interface. Nevertheless, it is probably a bad "example" to foist on people by not also following the established pattern of freeing the storage , so I'll make a modification for this shortly. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, March 12, 2010 11:47 PM To: m3commit at elegosoft.com; Coleburn, Randy Subject: RE: [M3commit] CVS Update: cm3 This change is suspicious. I think it leaks. Please see http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/os/WIN32/FSWin32.m3?rev=1.7;content-type=text%2Fplain for the pattern that is usually used. Granted, given the conservativeness of the collector -- it finds stuff in registers and the stack without a need for metadata telling it where the points are, and the fact that we no longer have VM-synchronization, I don't understand the need for the usual pattern. Maybe because it is willing to move stuff? Therefore a non-traced copy is made for C that won't ever move? - Jay > Date: Sat, 13 Mar 2010 02:11:24 +0000 > To: m3commit at elegosoft.com > From: rcoleburn at elego.de > Subject: [M3commit] CVS Update: cm3 > > CVSROOT: /usr/cvs > Changes by: rcoleburn at birch. 10/03/13 02:11:24 > > Modified files: > cm3/examples/calling-c-win32/src/: OK.m3 > > Log message: > M3toC.TtoS has been renamed to M3toC.SharedTtoS, so make adjustments to Ok.m3 so that it will compile.--R.Coleburn > ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 13 11:19:21 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 13 Mar 2010 10:19:21 +0000 Subject: [M3devel] comparisons vs. subranges Message-ID: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 13 11:49:37 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 13 Mar 2010 10:49:37 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: Message-ID: I might be able to do this. Give me a day or so.. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu; m3devel at elegosoft.com Date: Sat, 13 Mar 2010 10:19:21 +0000 Subject: [M3devel] comparisons vs. subranges <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Sat Mar 13 19:29:24 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Sat, 13 Mar 2010 13:29:24 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: Message-ID: <20100313182924.GA14329@topoi.pooq.com> On Sat, Mar 13, 2010 at 10:19:21AM +0000, Jay K wrote: > > <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; > <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; > > > > > Seems to me, the front end should notice these. > > The first should always be TRUE. > > And possibly, possibly warn. > > The second should always be FALSE. > > And possibly, possibly warn. > > > > "Generic" programming often hits this sort of thing though, a good reason not to warn. > > Programmer might also be working with existing code and have changed INTEGER to CARDINAL. > > Or be defending against future maintainers changing CARDINAL to INTEGER. Wasn't there a discussion a while ago about subranges out-of-bounds not being a safety problem? (Or was it arithmetic overflow?) This optimisation might well cause a quite hard-to-find bug if we don't guarantee subrange integrity. -- hendrik From jay.krell at cornell.edu Sat Mar 13 17:57:27 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 13 Mar 2010 16:57:27 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <20100313182924.GA14329@topoi.pooq.com> References: , <20100313182924.GA14329@topoi.pooq.com> Message-ID: Integer overflow is not a safety problem. That was the news (to me). Subranges do need to be enforced, at certain points, for their to be safety. This change doesn't change that. (Compiler bugs break safety, as always.) - Jay > Date: Sat, 13 Mar 2010 13:29:24 -0500 > From: hendrik at topoi.pooq.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] comparisons vs. subranges > > On Sat, Mar 13, 2010 at 10:19:21AM +0000, Jay K wrote: > > > > <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; > > <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; > > > > > > > > > > Seems to me, the front end should notice these. > > > > The first should always be TRUE. > > > > And possibly, possibly warn. > > > > The second should always be FALSE. > > > > And possibly, possibly warn. > > > > > > > > "Generic" programming often hits this sort of thing though, a good reason not to warn. > > > > Programmer might also be working with existing code and have changed INTEGER to CARDINAL. > > > > Or be defending against future maintainers changing CARDINAL to INTEGER. > > Wasn't there a discussion a while ago about subranges out-of-bounds not > being a safety problem? (Or was it arithmetic overflow?) This > optimisation might well cause a quite hard-to-find bug if we don't > guarantee subrange integrity. > > -- hendrik -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sat Mar 13 19:03:18 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sat, 13 Mar 2010 13:03:18 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: Message-ID: Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: > <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; > <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; > > > Seems to me, the front end should notice these. > The first should always be TRUE. > And possibly, possibly warn. > The second should always be FALSE. > And possibly, possibly warn. > > "Generic" programming often hits this sort of thing though, a good reason not to warn. > Programmer might also be working with existing code and have changed INTEGER to CARDINAL. > Or be defending against future maintainers changing CARDINAL to INTEGER. > > > The backend isn't give enough information, because CARDINAL = INTEGER as far > as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. > The only way to get unsigned types is to use ADDRESS from what I see. > > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 13 19:26:34 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 13 Mar 2010 18:26:34 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: , Message-ID: Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. What is the point of these functions Fold? Why does it have these various "cost" computations? Granted, they are rough. Should we beef up m3middle? m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. m3back is correct or extremely close to correct as far as I know. It is missing atomic operations for 8, 16, 64 bit operands. (PPC32 seems unable to support 64bit atomics btw.) I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. - Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 13:03:18 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] comparisons vs. subranges Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sat Mar 13 20:23:33 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sat, 13 Mar 2010 14:23:33 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: , Message-ID: On 13 Mar 2010, at 13:26, Jay K wrote: > Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. > > > What is the point of these functions Fold? So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. > Why does it have these various "cost" computations? Which cost computations are you referring to? > Granted, they are rough. > > > Should we beef up m3middle? That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. > m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. > It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. > It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. Designing a decent optimising middle-end takes significant time and effort, and will require more thought. > m3back is correct or extremely close to correct as far as I know. > It is missing atomic operations for 8, 16, 64 bit operands. > (PPC32 seems unable to support 64bit atomics btw.) Correct. > I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. > > > I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Sat, 13 Mar 2010 13:03:18 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] comparisons vs. subranges > > Jay, > > I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 13 Mar 2010, at 05:19, Jay K wrote: > > <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; > <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; > > > Seems to me, the front end should notice these. > The first should always be TRUE. > And possibly, possibly warn. > The second should always be FALSE. > And possibly, possibly warn. > > "Generic" programming often hits this sort of thing though, a good reason not to warn. > Programmer might also be working with existing code and have changed INTEGER to CARDINAL. > Or be defending against future maintainers changing CARDINAL to INTEGER. > > > The backend isn't give enough information, because CARDINAL = INTEGER as far > as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. > The only way to get unsigned types is to use ADDRESS from what I see. > > > - Jay > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sat Mar 13 20:45:29 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sat, 13 Mar 2010 14:45:29 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: , Message-ID: <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Jay, I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 14:23, Tony Hosking wrote: > On 13 Mar 2010, at 13:26, Jay K wrote: > >> Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. >> >> >> What is the point of these functions Fold? > > So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. > >> Why does it have these various "cost" computations? > > Which cost computations are you referring to? > >> Granted, they are rough. >> >> >> Should we beef up m3middle? > > That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. > >> m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. >> It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. >> It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. > > Designing a decent optimising middle-end takes significant time and effort, and will require more thought. > >> m3back is correct or extremely close to correct as far as I know. >> It is missing atomic operations for 8, 16, 64 bit operands. >> (PPC32 seems unable to support 64bit atomics btw.) > > Correct. > >> I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. >> >> >> I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. >> >> >> - Jay >> >> >> From: hosking at cs.purdue.edu >> Date: Sat, 13 Mar 2010 13:03:18 -0500 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] comparisons vs. subranges >> >> Jay, >> >> I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. >> >> Antony Hosking | Associate Professor | Computer Science | Purdue University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> On 13 Mar 2010, at 05:19, Jay K wrote: >> >> <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; >> <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; >> >> >> Seems to me, the front end should notice these. >> The first should always be TRUE. >> And possibly, possibly warn. >> The second should always be FALSE. >> And possibly, possibly warn. >> >> "Generic" programming often hits this sort of thing though, a good reason not to warn. >> Programmer might also be working with existing code and have changed INTEGER to CARDINAL. >> Or be defending against future maintainers changing CARDINAL to INTEGER. >> >> >> The backend isn't give enough information, because CARDINAL = INTEGER as far >> as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. >> The only way to get unsigned types is to use ADDRESS from what I see. >> >> >> - Jay >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmuysers at hotmail.com Sat Mar 13 23:20:07 2010 From: dmuysers at hotmail.com (Dirk Muysers) Date: Sat, 13 Mar 2010 23:20:07 +0100 Subject: [M3devel] bandwith Message-ID: I don't know if other people have the same feeling, but I think that Messrs. Krell and Hosking should keep their polemics to themselves and let the community partake only in the result of a constructive consensus. As a USER of the m3 compiler on windows, I find the recent developments scary, so I prefer to stick to my installed rather old release (5.3.1 if memory serves, BTW the version number should figure at a prominent place in every release.) I am afraid that M3 is going the way of fubar and I am slowly loosing interest. To make it work for all these different platforms seems a herculean task and rather pointless endeavour. Why not use all that energy to try a radically disruptive strategy such as a platform agnostic intermediate form targeting a JIT. Let me apologize in advance to the whole community for the present fit of an old men's anger. d. muysers -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Sun Mar 14 02:54:41 2010 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sun, 14 Mar 2010 01:54:41 +0000 (GMT) Subject: [M3devel] bandwith In-Reply-To: Message-ID: <562452.30843.qm@web23608.mail.ird.yahoo.com> Hi all: I think is an interesting topic, however I disagree with you in the old version mentioned because since then CM3 has evolved a lot in threading, garbage collection and gcc based backend packages, just to name a few. I however seem to recall some discussions, but unfortunately there wasn't sometimes an ending point (i.e my discussion), however without more participation it really becomes boring at one point I expect to be fully honest I do have fun compiling things but probably the community lacks more understanding of the system itself or of the current issues to work in that which needs to be done. Track system by itself is great but probably we haven't used yet in its full potential and that's the reason I waited for so long to report things that would be useful before in ticket trac system (nor I did have the idea to ask you). Current situation in this matter is going better I think. Therefore there could be plenty of opportunities of things to work on in a fashionable way if we may. I don't know but I think the current stability should be the worrisome and the history of this could go back to C interfaces and son on. I haven't asked yet where are the Unix INTERFACE constants and members which are need for packages that before where compiling that use this low-level style like browsers webcard, postcard, Deckscape and Webscape and m3-mail packages and m3-lectern which I could get to compile some time before this Unix changes. However I know that some bugs they had I expect to be gone because this changes brought many other improvements to overall programming environment. Besides this, software engineering tools could be made aware of the project like this one done for pm3: http://libresoft.dat.escet.urjc.es/cvsanal/pm3-cvs/index.php?menu=Index (the manual of it on http://gsyc.es/~carlosgc/files/cvsanaly.pdf software on http://tools.libresoft.es/cvsanaly?rev=1248394389) I know there is also a Modula-3 aware re engineering tools it was done by Peter Klien at Aachen University I know there was supossed to be a working prototype at some point available for download http://pi.informatik.uni-siegen.de/stt/23_3/05_Dissertationen/klein.pdf This kind of tools could be managed to get integrated with M3clipse plugin work done by Bert Laverman which just lacks semantic analysis, and cm3ide which may bundle cvsup and or dcvs as a way of distributed software configuration management and development environment perhaps with Elego Compact (and don't forget Obliq packages, I remember a Professor told me that it competed at some point with JavaScript to be scripting language for the web. It has even type inference algorithm now so who knows why isn't used in our community, I know I will try to get into it but it needs time; there are some examples in Professor's courses that could help to this, besides other things of M3 system level demo applications too) Thanks for your comments I hope this helps a bit --- El s?b, 13/3/10, Dirk Muysers escribi?: > De: Dirk Muysers > Asunto: [M3devel] bandwith > Para: m3devel at elegosoft.com > Fecha: s?bado, 13 de marzo, 2010 17:20 > > > > > > > > I don't know if other > people have the same feeling, > but I think that Messrs. Krell and Hosking > should keep their polemics to > themselves and let the > community partake only > in the result of > a constructive consensus. > > > As a USER of the m3 > compiler on windows, I > find the recent developments scary, so I prefer > to > stick to my installed > rather old release (5.3.1 if > memory serves, BTW the version number should > figure at a prominent > place in every > release.) I am afraid that M3 is going the way of fubar and > I > am slowly loosing > interest. > > To make it work for all > these different platforms > seems a herculean task > and rather > pointless > endeavour. Why not use all > that energy to > try a radically > disruptive strategy such as > a platform > agnostic intermediate form > targeting a > JIT. > > Let > me apologize in advance to the whole > community for the present fit of an old men's > anger. > d. > muysers > From jay.krell at cornell.edu Sun Mar 14 04:52:15 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 03:52:15 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: Hm. You mean, like I'm not supposed to be able to say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (a < b) => IO.Put("true\n"); | (a > b) => IO.Put("false\n"); END; but I can say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); END; (I'd like to be able to say FIRST(a), etc., but I don't think it is allowed.) "constants" kind of seem like an optimization themselves. Other than efficiency concerns, they could all be variables initialized by module initializers, that the frontend doesn't let you write to. Including de-optimizations like runtime allocation/sizing of arrays based on const/non-const sizes. Look at how I changed const to var already in m3core/unix, and then had to change CASE to if/elseif ladders. It's a minor matter of what the language lets you say, among various basically equivalent forms. The compiler could allow case to target non-constants. It'd just mainly be slower. There is the matter of CASE arms can't overlap, where if/elseif ladders can. Because CASE is conceptually compared all at once where if/elseif is sequential. Ok. I see a few options at least. m3middle could look basically like m3front, but with various checks like type checking removed and assumed true. This would be an arduous task of duplication. M3CG.i3 Var and Target.Int could gain the following fields (or methods): min_valid := FALSE; max_valid := FALSE; min := TInt.Zero; max := TInt.Zero; m3front could fill them in a few cases. Initially none, but then a few as they are discovered to be easy, correct, efficient. For constants, min = max value. For subranges, min/max come the type. The backends could use them (if valid) and possibly transform them. e.g. MIN(a,b) with valid min/max yields an expression where min is the min of the two mins, max is the min of the two maxes. MIN([0..1],[2..3]) < 2 => TRUE. Probably more correct would be: PROCEDURE declare_enum (xx: T; t: TypeUID; n_elts: INTEGER; s: BitSize) = PROCEDURE declare_subrange (xx: T; t, domain: TypeUID; READONLY min, max: Target.Int; s: BitSize) => already exist and then PROCEDURE load (xx: T; v: Var; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_indirect (xx: T; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_integer (xx: T; t: IType; READONLY i: Target.Int) = PROCEDURE load_ordered (xx: T; t: MType; u: ZType; order: MemoryOrder) = etc. should all take an optional TypeUID. The second proposal I've thought a bit more about and it seems very easy. The last idea seems cleaner, but with possibly signifiant downside in that e.g. Word.T is not a subrange. There's also some unclarity with respect to Word.LT(expression, -1); -1 is a valid Word interpreted as a large number. One would want to be careful not to break that. I believe this is related to what you were saying, where operations are typed, not values. It could be that every operation needs min/max or typeuid (pairs of them for binary operations). Can we take a stab at something like this? This might also serve to replace some of what m3back does with constants already, since constants would fall out of subranges of length 1. Thanks, -Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 14:45:29 -0500 To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com; jay.krell at cornell.edu Subject: Re: [M3devel] comparisons vs. subranges Jay, I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 14:23, Tony Hosking wrote: On 13 Mar 2010, at 13:26, Jay K wrote: Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. What is the point of these functions Fold? So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. Why does it have these various "cost" computations? Which cost computations are you referring to? Granted, they are rough. Should we beef up m3middle? That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. Designing a decent optimising middle-end takes significant time and effort, and will require more thought. m3back is correct or extremely close to correct as far as I know. It is missing atomic operations for 8, 16, 64 bit operands. (PPC32 seems unable to support 64bit atomics btw.) Correct. I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. - Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 13:03:18 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] comparisons vs. subranges Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 05:31:58 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 04:31:58 +0000 Subject: [M3devel] bandwith In-Reply-To: <562452.30843.qm@web23608.mail.ird.yahoo.com> References: , <562452.30843.qm@web23608.mail.ird.yahoo.com> Message-ID: Arg. Change does not equal instability. Status quo does not equal better. Do you want 64bit LONGINT? Maybe, maybe not. Do you want every integer comparison to needlessly use a temporary on the stack and extra instructions? Maybe, maybe not. Do you want the tables in hand.c statically linked? Maybe, maybe not. They caused quite a headache albeit briefly, when we weren't statically linking them. Standalone apps worked fine, dynamically linked exhibited odd behavior. The odd behavior was "just" gui drawn incorrectly, but was actually indicative of deep problems. Do you want dead buggy code sitting around, waiting to be silently used as m3front changes? (e.g. some forms of extract look wrong). A giant lock in ThreadWin32? vs. the simple pattern documented on the web and what Java uses? Granted, I removed the threadpool, maybe too much simplicity and not enough efficiency? (But have you noticed how much ThreadWin32.m3 did churn through the years anyway?) Can you point to actual bugs? Granted, I don't have specifics in many cases either, such as the bad perf of the giant lock, but everyone knows about them. The system builds itself. I probably rest too much on that, granted. m3-sys/m3tests does pretty well (some stuff does need looking at). I try to add more test coverage when I make changes, but I am lazy, granted. Some non-zero level of progress is presumably desired. The Unix interface was painful to maintain and buggy and tended toward unsafety but claiming safety. It is much improved now. Much safer, much easier to port. A little less efficient. Sort off less amenable to cross building, though really hand.c and lack of an assembler and linker are enough to kill cross builds. The system has never as far I know been really cross buildable, without gcc/binutils support -- has always needed an assembler and linker, and usually needed a C compiler. (NT386 doesn't need an assembler). Just because interface Unix allowed for compiling more code in the past did not make it better or more correct. In fact possibly the opposite. Granted, we should try have it all -- correctness, features, maintability, efficiency. There was a lot of cruft in there getting copied from port to port without being revisited. Stuff having to do with Linux 1.x was in various ports. Every major version of FreeBSD implied extra work. We had FreeBSD, FreeBSD2, FreeBSD3, FreeBSD4. (There are still problems here actually, FreeBSD7 64bit I believe broke the ABI in networking, and we still have the related structs I think -- there is still bad stuff in interface Unix, just a lot less now. I still want to whittle it down more.) There is still code in the system that redeclares struct tm incorrectly, attempting to workaround interface Unix deficiency, when in fact interface Unix was correctly limited (the fields at the end, and perhaps the order of the fields). In the web browsers as I recall (which never seemed to work much in the first place, but they did/do build and come up). > > agnostic intermediate form targeting a JIT. Other than NT386, our portability rests on gcc and Posix, which are very portable with very little effort. I haven't seen any JIT that is as portable. For example there are a few systems we can't run Hudson on because it uses Java. e.g. MIPS64_OPENBSD, PPC32_OPENBSD I would actually like to generate C to gain even more portability. Granted, JIT has the nice property of target-independent distributable binaries. "Package building" matrix could be cut down. Generating C can get you target-independent source distributions, as most things use. (Granted, compilers are somewhat special, except that the world long ago got past not having a C compiler). Targeting a JIT will probably be way more work than anyone has put into this system in over a decade. Still, not a bad idea. It lets you reuse other people's code generators similarly to how we reuse gcc. Java has been made portable to any processor, running Linux. But not yet to other operating systems. I think without too much work, we could use our exiting IL that parse.c reads. It'd just need more context in places, to avoid a need for globals. You could just write an interpreter. It wouldn't be fast, but it would gain more portability probably. Probably not the best idea though. Also, the Java JIT, as one example, is among the more portable, and among the more difficult to target. It doesn't model a CPU as you might expect. It models more so the Java programming language. Which includes non-optional safety. You'd have better luck probably with mono/CLR, not sure as to the level of portability. Mono is pretty widely ported though. Really though, gcc+Posix provides a huge amount of portability with very little effort on our part. The larger work is really obtaining, housing, powering, installing the various systems. If you go the C/portable-JIT route, what you really do is you just stop testing the various targets, for better and worse, probably perfectly ok. Just as nobody tests hello world on every system. They just know it works because it is so portable. > webcard, postcard, Deckscape and Webscape and m3-mail packages and m3-lectern They don't compile? > > rather old release (5.3.1 if memory serves, You can keep using it? Maybe compare that version to current and see if there are particular problems? > > BTW the version number should figure at a prominent place in every release. cm3 -version? As to where Tony and I should argue, well, that's not a generally answerable question. I find that a conversation involving just two people not ideal, no matter who they are. Being under outside scrutiny increases civility, sometimes add more than two opinions, etc. I grant that m3 is such a small community that we don't have an m3-users vs. m3-devel group, and m3-devel is woefully small, and the two groups a bit inappropriately mixed. - Jay > Date: Sun, 14 Mar 2010 01:54:41 +0000 > From: dabenavidesd at yahoo.es > To: m3devel at elegosoft.com; dmuysers at hotmail.com > Subject: Re: [M3devel] bandwith > > Hi all: > I think is an interesting topic, however I disagree with you in the old version mentioned because since then CM3 has evolved a lot in threading, garbage collection and gcc based backend packages, just to name a few. > I however seem to recall some discussions, but unfortunately there wasn't sometimes an ending point (i.e my discussion), however without more participation it really becomes boring at one point I expect to be fully honest I do have fun compiling things but probably the community lacks more understanding of the system itself or of the current issues to work in that which needs to be done. Track system by itself is great but probably we haven't used yet in its full potential and that's the reason I waited for so long to report things that would be useful before in ticket trac system (nor I did have the idea to ask you). Current situation in this matter is going better I think. > Therefore there could be plenty of opportunities of things to work on in a fashionable way if we may. > I don't know but I think the current stability should be the worrisome and the history of this could go back to C interfaces and son on. I haven't asked yet where are the Unix INTERFACE constants and members which are need for packages that before where compiling that use this low-level style like browsers webcard, postcard, Deckscape and Webscape and m3-mail packages and m3-lectern which I could get to compile some time before this Unix changes. > However I know that some bugs they had I expect to be gone because this changes brought many other improvements to overall programming environment. > Besides this, software engineering tools could be made aware of the project like this one done for pm3: > http://libresoft.dat.escet.urjc.es/cvsanal/pm3-cvs/index.php?menu=Index > (the manual of it on http://gsyc.es/~carlosgc/files/cvsanaly.pdf software on http://tools.libresoft.es/cvsanaly?rev=1248394389) > I know there is also a Modula-3 aware re engineering tools it was done by Peter Klien at Aachen University I know there was supossed to be a working prototype at some point available for download > http://pi.informatik.uni-siegen.de/stt/23_3/05_Dissertationen/klein.pdf > This kind of tools could be managed to get integrated with M3clipse plugin work done by Bert Laverman which just lacks semantic analysis, and cm3ide which may bundle cvsup and or dcvs as a way of distributed software configuration management and development environment perhaps with Elego Compact (and don't forget Obliq packages, I remember a Professor told me that it competed at some point with JavaScript to be scripting language for the web. It has even type inference algorithm now so who knows why isn't used in our community, I know I will try to get into it but it needs time; there are some examples in Professor's courses that could help to this, besides other things of M3 system level demo applications too) > Thanks for your comments I hope this helps a bit > > > --- El s?b, 13/3/10, Dirk Muysers escribi?: > > > De: Dirk Muysers > > Asunto: [M3devel] bandwith > > Para: m3devel at elegosoft.com > > Fecha: s?bado, 13 de marzo, 2010 17:20 > > > > > > > > > > > > > > > > I don't know if other > > people have the same feeling, > > but I think that Messrs. Krell and Hosking > > should keep their polemics to > > themselves and let the > > community partake only > > in the result of > > a constructive consensus. > > > > > > As a USER of the m3 > > compiler on windows, I > > find the recent developments scary, so I prefer > > to > > stick to my installed > > rather old release (5.3.1 if > > memory serves, BTW the version number should > > figure at a prominent > > place in every > > release.) I am afraid that M3 is going the way of fubar and > > I > > am slowly loosing > > interest. > > > > To make it work for all > > these different platforms > > seems a herculean task > > and rather > > pointless > > endeavour. Why not use all > > that energy to > > try a radically > > disruptive strategy such as > > a platform > > agnostic intermediate form > > targeting a > > JIT. > > > > Let > > me apologize in advance to the whole > > community for the present fit of an old men's > > anger. > > d. > > muysers > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 06:01:24 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 05:01:24 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: Hm. How about elsewhere m3front? Optimizing middle end not necessarily a different package? (esp. given that m3front is already big enough to afford being organized into directories: m3front/src/middle, m3front/src/target-independent-optimizations?) Maybe elsewhere in the two files I changed? Prep or PrepBR or Compile instead of Fold? I don't understand this Prep vs. PrepBR business. I found out that "BR" stands for branch, but that doesn't explain it me. Also PrepBR and Compile look very similar. I also noticed that Fold gets called twice (I had some RTIO in it) but didn't look into why. Surely doing it in PrepBR or Compile is correct or CG.m3 is correct, reasonable, efficient, maintainable? I can see why Fold would be wrong -- allowing code to compile, with a reasonable meaning, but that isn't supposed to. Or in CG.m3, which is conceptually "the bottom" of m3front, any two adjacent pieces can be considered merged into one layer. I'll look at both of those options later. I think "subrange analysis" if done enough might yield efficiencies in real code. Though m3front might already do enough of it -- e.g. the fact that you can't modify a FOR loop index affords some efficiencies, that are probably already taken into account. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 03:52:15 +0000 Hm. You mean, like I'm not supposed to be able to say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (a < b) => IO.Put("true\n"); | (a > b) => IO.Put("false\n"); END; but I can say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); END; (I'd like to be able to say FIRST(a), etc., but I don't think it is allowed.) "constants" kind of seem like an optimization themselves. Other than efficiency concerns, they could all be variables initialized by module initializers, that the frontend doesn't let you write to. Including de-optimizations like runtime allocation/sizing of arrays based on const/non-const sizes. Look at how I changed const to var already in m3core/unix, and then had to change CASE to if/elseif ladders. It's a minor matter of what the language lets you say, among various basically equivalent forms. The compiler could allow case to target non-constants. It'd just mainly be slower. There is the matter of CASE arms can't overlap, where if/elseif ladders can. Because CASE is conceptually compared all at once where if/elseif is sequential. Ok. I see a few options at least. m3middle could look basically like m3front, but with various checks like type checking removed and assumed true. This would be an arduous task of duplication. M3CG.i3 Var and Target.Int could gain the following fields (or methods): min_valid := FALSE; max_valid := FALSE; min := TInt.Zero; max := TInt.Zero; m3front could fill them in a few cases. Initially none, but then a few as they are discovered to be easy, correct, efficient. For constants, min = max value. For subranges, min/max come the type. The backends could use them (if valid) and possibly transform them. e.g. MIN(a,b) with valid min/max yields an expression where min is the min of the two mins, max is the min of the two maxes. MIN([0..1],[2..3]) < 2 => TRUE. Probably more correct would be: PROCEDURE declare_enum (xx: T; t: TypeUID; n_elts: INTEGER; s: BitSize) = PROCEDURE declare_subrange (xx: T; t, domain: TypeUID; READONLY min, max: Target.Int; s: BitSize) => already exist and then PROCEDURE load (xx: T; v: Var; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_indirect (xx: T; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_integer (xx: T; t: IType; READONLY i: Target.Int) = PROCEDURE load_ordered (xx: T; t: MType; u: ZType; order: MemoryOrder) = etc. should all take an optional TypeUID. The second proposal I've thought a bit more about and it seems very easy. The last idea seems cleaner, but with possibly signifiant downside in that e.g. Word.T is not a subrange. There's also some unclarity with respect to Word.LT(expression, -1); -1 is a valid Word interpreted as a large number. One would want to be careful not to break that. I believe this is related to what you were saying, where operations are typed, not values. It could be that every operation needs min/max or typeuid (pairs of them for binary operations). Can we take a stab at something like this? This might also serve to replace some of what m3back does with constants already, since constants would fall out of subranges of length 1. Thanks, -Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 14:45:29 -0500 To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com; jay.krell at cornell.edu Subject: Re: [M3devel] comparisons vs. subranges Jay, I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 14:23, Tony Hosking wrote: On 13 Mar 2010, at 13:26, Jay K wrote: Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. What is the point of these functions Fold? So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. Why does it have these various "cost" computations? Which cost computations are you referring to? Granted, they are rough. Should we beef up m3middle? That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. Designing a decent optimising middle-end takes significant time and effort, and will require more thought. m3back is correct or extremely close to correct as far as I know. It is missing atomic operations for 8, 16, 64 bit operands. (PPC32 seems unable to support 64bit atomics btw.) Correct. I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. - Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 13:03:18 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] comparisons vs. subranges Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 06:15:22 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 05:15:22 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: > Prep vs. PrepBR vs. Compile Expr.i3 documents it. (*** phase 4 ****) (* Expressions are compiled in two steps: Prep: emit any code that includes branchs or stores Compile: emit the rest of the code *) PROCEDURE Prep (t: T); PROCEDURE Compile (t: T); (* emits code to evaluate the expression onto the top of stack *) PROCEDURE PrepLValue (t: T; traced: BOOLEAN); PROCEDURE CompileLValue (t: T; traced: BOOLEAN); (* emits code to evaluate 't's L-value into s0.A. *) PROCEDURE CompileAddress (t: T; traced: BOOLEAN); (* emits code to evaluate 't's byte address onto the top of stack. Use PrepLValue to prep these expressions. *) PROCEDURE PrepBranch (t: T; true, false: CG.Label; freq: CG.Frequency); PROCEDURE CompileBranch (t: T; true, false: CG.Label; freq: CG.Frequency); (* emits code to evaluate the expression and conditionally branch to 'true' or 'false' depending on its boolean value. 'freq' is the estimated frequency that the specified branch will be taken. *) - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 05:01:24 +0000 Hm. How about elsewhere m3front? Optimizing middle end not necessarily a different package? (esp. given that m3front is already big enough to afford being organized into directories: m3front/src/middle, m3front/src/target-independent-optimizations?) Maybe elsewhere in the two files I changed? Prep or PrepBR or Compile instead of Fold? I don't understand this Prep vs. PrepBR business. I found out that "BR" stands for branch, but that doesn't explain it me. Also PrepBR and Compile look very similar. I also noticed that Fold gets called twice (I had some RTIO in it) but didn't look into why. Surely doing it in PrepBR or Compile is correct or CG.m3 is correct, reasonable, efficient, maintainable? I can see why Fold would be wrong -- allowing code to compile, with a reasonable meaning, but that isn't supposed to. Or in CG.m3, which is conceptually "the bottom" of m3front, any two adjacent pieces can be considered merged into one layer. I'll look at both of those options later. I think "subrange analysis" if done enough might yield efficiencies in real code. Though m3front might already do enough of it -- e.g. the fact that you can't modify a FOR loop index affords some efficiencies, that are probably already taken into account. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 03:52:15 +0000 Hm. You mean, like I'm not supposed to be able to say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (a < b) => IO.Put("true\n"); | (a > b) => IO.Put("false\n"); END; but I can say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); END; (I'd like to be able to say FIRST(a), etc., but I don't think it is allowed.) "constants" kind of seem like an optimization themselves. Other than efficiency concerns, they could all be variables initialized by module initializers, that the frontend doesn't let you write to. Including de-optimizations like runtime allocation/sizing of arrays based on const/non-const sizes. Look at how I changed const to var already in m3core/unix, and then had to change CASE to if/elseif ladders. It's a minor matter of what the language lets you say, among various basically equivalent forms. The compiler could allow case to target non-constants. It'd just mainly be slower. There is the matter of CASE arms can't overlap, where if/elseif ladders can. Because CASE is conceptually compared all at once where if/elseif is sequential. Ok. I see a few options at least. m3middle could look basically like m3front, but with various checks like type checking removed and assumed true. This would be an arduous task of duplication. M3CG.i3 Var and Target.Int could gain the following fields (or methods): min_valid := FALSE; max_valid := FALSE; min := TInt.Zero; max := TInt.Zero; m3front could fill them in a few cases. Initially none, but then a few as they are discovered to be easy, correct, efficient. For constants, min = max value. For subranges, min/max come the type. The backends could use them (if valid) and possibly transform them. e.g. MIN(a,b) with valid min/max yields an expression where min is the min of the two mins, max is the min of the two maxes. MIN([0..1],[2..3]) < 2 => TRUE. Probably more correct would be: PROCEDURE declare_enum (xx: T; t: TypeUID; n_elts: INTEGER; s: BitSize) = PROCEDURE declare_subrange (xx: T; t, domain: TypeUID; READONLY min, max: Target.Int; s: BitSize) => already exist and then PROCEDURE load (xx: T; v: Var; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_indirect (xx: T; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_integer (xx: T; t: IType; READONLY i: Target.Int) = PROCEDURE load_ordered (xx: T; t: MType; u: ZType; order: MemoryOrder) = etc. should all take an optional TypeUID. The second proposal I've thought a bit more about and it seems very easy. The last idea seems cleaner, but with possibly signifiant downside in that e.g. Word.T is not a subrange. There's also some unclarity with respect to Word.LT(expression, -1); -1 is a valid Word interpreted as a large number. One would want to be careful not to break that. I believe this is related to what you were saying, where operations are typed, not values. It could be that every operation needs min/max or typeuid (pairs of them for binary operations). Can we take a stab at something like this? This might also serve to replace some of what m3back does with constants already, since constants would fall out of subranges of length 1. Thanks, -Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 14:45:29 -0500 To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com; jay.krell at cornell.edu Subject: Re: [M3devel] comparisons vs. subranges Jay, I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 14:23, Tony Hosking wrote: On 13 Mar 2010, at 13:26, Jay K wrote: Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. What is the point of these functions Fold? So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. Why does it have these various "cost" computations? Which cost computations are you referring to? Granted, they are rough. Should we beef up m3middle? That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. Designing a decent optimising middle-end takes significant time and effort, and will require more thought. m3back is correct or extremely close to correct as far as I know. It is missing atomic operations for 8, 16, 64 bit operands. (PPC32 seems unable to support 64bit atomics btw.) Correct. I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. - Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 13:03:18 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] comparisons vs. subranges Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 11:43:57 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 10:43:57 +0000 Subject: [M3devel] comparing different types/signedness Message-ID: Tony, is it deliberate in m3front/src/misc/cg.m3 procedure compare that we are often asked to compare different types, even different signedness? It seems a little unusual. I think it really might help to extend TInt to 9 or more bytes so that any pair of values we "really" encounter can be extended to a common type/precision. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 13:36:50 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 12:36:50 +0000 Subject: [M3devel] comparing different types/signedness In-Reply-To: References: Message-ID: I got through my problem here, not a big deal. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu; m3devel at elegosoft.com Date: Sun, 14 Mar 2010 10:43:57 +0000 Subject: [M3devel] comparing different types/signedness Tony, is it deliberate in m3front/src/misc/cg.m3 procedure compare that we are often asked to compare different types, even different signedness? It seems a little unusual. I think it really might help to extend TInt to 9 or more bytes so that any pair of values we "really" encounter can be extended to a common type/precision. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 13:43:02 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 12:43:02 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: Tony how about the attached? It achieves the "same" thing as before. CG is about the lowest level in m3front, therefore akin to any middle end below m3front. The vast bulk of the change is in CG.m3, with a small change in Variable.m3 to pass it the bounds. It took me a while to cope with the mixture of signed and unsigned that the front end throws at its CG.m3. More can be done here. e.g. mod a positive non-zero constant returns 0..n-1. min and max(a,b) has bounds computable from the bounds of a and b. abs returns a positive number, except for the overflow case neg(abs) returns 0 or negative (again with some chance of overflow) I think the change is ok. There is the small matter of how to call GetBounds. I find it a little wierd that some versions of GetBounds return a boolean, some do not. The signed/unsigned cases could be combined down somehow, replacing min/max in some places with zero. Load_addr_of_temp should probably use WITH. Other forms of Load e.g. Load_indirect should probably be changed analogously. There is also a matter of "bounds" for non-ordinal types. For example a set might be a constant. It might be possible to reason about floating point. But I didn't do any this stuf. Just ordinal types. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 05:15:22 +0000 > Prep vs. PrepBR vs. Compile Expr.i3 documents it. (*** phase 4 ****) (* Expressions are compiled in two steps: Prep: emit any code that includes branchs or stores Compile: emit the rest of the code *) PROCEDURE Prep (t: T); PROCEDURE Compile (t: T); (* emits code to evaluate the expression onto the top of stack *) PROCEDURE PrepLValue (t: T; traced: BOOLEAN); PROCEDURE CompileLValue (t: T; traced: BOOLEAN); (* emits code to evaluate 't's L-value into s0.A. *) PROCEDURE CompileAddress (t: T; traced: BOOLEAN); (* emits code to evaluate 't's byte address onto the top of stack. Use PrepLValue to prep these expressions. *) PROCEDURE PrepBranch (t: T; true, false: CG.Label; freq: CG.Frequency); PROCEDURE CompileBranch (t: T; true, false: CG.Label; freq: CG.Frequency); (* emits code to evaluate the expression and conditionally branch to 'true' or 'false' depending on its boolean value. 'freq' is the estimated frequency that the specified branch will be taken. *) - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 05:01:24 +0000 Hm. How about elsewhere m3front? Optimizing middle end not necessarily a different package? (esp. given that m3front is already big enough to afford being organized into directories: m3front/src/middle, m3front/src/target-independent-optimizations?) Maybe elsewhere in the two files I changed? Prep or PrepBR or Compile instead of Fold? I don't understand this Prep vs. PrepBR business. I found out that "BR" stands for branch, but that doesn't explain it me. Also PrepBR and Compile look very similar. I also noticed that Fold gets called twice (I had some RTIO in it) but didn't look into why. Surely doing it in PrepBR or Compile is correct or CG.m3 is correct, reasonable, efficient, maintainable? I can see why Fold would be wrong -- allowing code to compile, with a reasonable meaning, but that isn't supposed to. Or in CG.m3, which is conceptually "the bottom" of m3front, any two adjacent pieces can be considered merged into one layer. I'll look at both of those options later. I think "subrange analysis" if done enough might yield efficiencies in real code. Though m3front might already do enough of it -- e.g. the fact that you can't modify a FOR loop index affords some efficiencies, that are probably already taken into account. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 03:52:15 +0000 Hm. You mean, like I'm not supposed to be able to say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (a < b) => IO.Put("true\n"); | (a > b) => IO.Put("false\n"); END; but I can say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); END; (I'd like to be able to say FIRST(a), etc., but I don't think it is allowed.) "constants" kind of seem like an optimization themselves. Other than efficiency concerns, they could all be variables initialized by module initializers, that the frontend doesn't let you write to. Including de-optimizations like runtime allocation/sizing of arrays based on const/non-const sizes. Look at how I changed const to var already in m3core/unix, and then had to change CASE to if/elseif ladders. It's a minor matter of what the language lets you say, among various basically equivalent forms. The compiler could allow case to target non-constants. It'd just mainly be slower. There is the matter of CASE arms can't overlap, where if/elseif ladders can. Because CASE is conceptually compared all at once where if/elseif is sequential. Ok. I see a few options at least. m3middle could look basically like m3front, but with various checks like type checking removed and assumed true. This would be an arduous task of duplication. M3CG.i3 Var and Target.Int could gain the following fields (or methods): min_valid := FALSE; max_valid := FALSE; min := TInt.Zero; max := TInt.Zero; m3front could fill them in a few cases. Initially none, but then a few as they are discovered to be easy, correct, efficient. For constants, min = max value. For subranges, min/max come the type. The backends could use them (if valid) and possibly transform them. e.g. MIN(a,b) with valid min/max yields an expression where min is the min of the two mins, max is the min of the two maxes. MIN([0..1],[2..3]) < 2 => TRUE. Probably more correct would be: PROCEDURE declare_enum (xx: T; t: TypeUID; n_elts: INTEGER; s: BitSize) = PROCEDURE declare_subrange (xx: T; t, domain: TypeUID; READONLY min, max: Target.Int; s: BitSize) => already exist and then PROCEDURE load (xx: T; v: Var; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_indirect (xx: T; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_integer (xx: T; t: IType; READONLY i: Target.Int) = PROCEDURE load_ordered (xx: T; t: MType; u: ZType; order: MemoryOrder) = etc. should all take an optional TypeUID. The second proposal I've thought a bit more about and it seems very easy. The last idea seems cleaner, but with possibly signifiant downside in that e.g. Word.T is not a subrange. There's also some unclarity with respect to Word.LT(expression, -1); -1 is a valid Word interpreted as a large number. One would want to be careful not to break that. I believe this is related to what you were saying, where operations are typed, not values. It could be that every operation needs min/max or typeuid (pairs of them for binary operations). Can we take a stab at something like this? This might also serve to replace some of what m3back does with constants already, since constants would fall out of subranges of length 1. Thanks, -Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 14:45:29 -0500 To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com; jay.krell at cornell.edu Subject: Re: [M3devel] comparisons vs. subranges Jay, I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 14:23, Tony Hosking wrote: On 13 Mar 2010, at 13:26, Jay K wrote: Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. What is the point of these functions Fold? So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. Why does it have these various "cost" computations? Which cost computations are you referring to? Granted, they are rough. Should we beef up m3middle? That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. Designing a decent optimising middle-end takes significant time and effort, and will require more thought. m3back is correct or extremely close to correct as far as I know. It is missing atomic operations for 8, 16, 64 bit operands. (PPC32 seems unable to support 64bit atomics btw.) Correct. I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. - Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 13:03:18 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] comparisons vs. subranges Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 1.txt URL: From rodney_bates at lcwb.coop Sun Mar 14 16:20:29 2010 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 14 Mar 2010 10:20:29 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <20100313182924.GA14329@topoi.pooq.com> References: <20100313182924.GA14329@topoi.pooq.com> Message-ID: <4B9CFEBD.8050309@lcwb.coop> hendrik at topoi.pooq.com wrote: > > Wasn't there a discussion a while ago about subranges out-of-bounds not > being a safety problem? (Or was it arithmetic overflow?) This > optimisation might well cause a quite hard-to-find bug if we don't > guarantee subrange integrity. Subrange is a safety problem. Overflow is not, although it can be a valuable way for the language/implementation to help find bugs. Most of the definitions of safety are neither very useful nor consistent with common, informal usage. My definition is that safety means everything that can happen can be explained and understood using only the concepts of the programming language. You don't have to resort to knowing about machine level stuff, especially bit representations and the fact that things that are entirely autonomous in the language (e.g. separately declared variables) actually occupy the same homogeneous memory, along with machine code and all sorts of other stuff. So, if LAST(INTEGER)+LAST(INTEGER) overflows, even if it is undefined by the language what happens, all the possibilities are still comprehensible in terms of the language. Either you get some value that is a member of INTEGER, or an exception, all of which are language concepts. But if you could assign 16_FF to a variable of type [0..20], you get something that can only be understood by looking at the machine-level representation. Most likely, it is a bit pattern that would represent a value that is not a member of the variable's type. This is a safely problem. BTW, it's also implementation- and target-dependent. > > -- hendrik > From rodney_bates at lcwb.coop Sun Mar 14 16:58:21 2010 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 14 Mar 2010 10:58:21 -0500 Subject: [M3devel] bandwith In-Reply-To: References: Message-ID: <4B9D079D.1060207@lcwb.coop> Dirk Muysers wrote: > > To make it work for all these different platforms seems a herculean task > and rather pointless > endeavour. Why not use all that energy to try a radically disruptive > strategy such as a platform > agnostic intermediate form targeting a JIT. > I have been a little way down this road at least three times, with at least two different languages, and it turns out not to be what it would appear at first glance, at least for the JVM. Java is a mostly object-oriented language, meaning the non-object-oriented types and operations are very limited. Most of the interesting stuff can be done only with full-blown heap allocated objects and dispatching methods. Even the equivalents of records and arrays are this way inb Java. When you don't need this full generality and power, you pay big performance penalties for heap allocation, and this is worse and much deeper than the performance penalties of an interpretive implementation. (which, of course, a JIT code generator partially avoids, and a traditional source-to-machine compiler avoids altogether, all with no language changes.) Many languages, Modula-3 included, give this generality for when you need it, but also give you lighter weight alternatives, for use when appropriate. So when you try to implement some other language using JVM, you soon discover that JVM lacks a lot of operations needed for the not-so- impoverished non-object-oriented types of the language. Whether or how badly the .net/mono virtual machine suffers from this, I don't know. It apparently was designed to support a much wider range of languages, but I seem to recall hearing some discussions that suggested it had some of the same limitations. Of course, we could invent our own virtual machine (M-code? MVM?) ;-). This whole approach is of course, not very suitable for systems programming, one of Modula-3's strengths. > d. muysers From rodney_bates at lcwb.coop Sun Mar 14 17:54:58 2010 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 14 Mar 2010 11:54:58 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: <4B9D14E2.6020403@lcwb.coop> Jay K wrote: > Hm. You mean, like I'm not supposed to be able to say: > > > PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= > BEGIN > CASE bool OF > | (a < b) => IO.Put("true\n"); > | (a > b) => IO.Put("false\n"); > END; > No, because a and b are not constants, and thus neither is a < b. Use an IF ladder for this. > > but I can say: > > > PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= > BEGIN > CASE bool OF > | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); > | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); > END; Yes. You can apply FIRST to a(non-open) array or ordinal _type_. > > (I'd like to be able to say FIRST(a), etc., but I don't think it is > allowed.) > Yes, or rather no. Yes, it's not allowed. Yes, we have no bananas. You can apply FIRST to a _variable_ of array type and it will be legal and (if not an open array), constant. You can't apply FIRST to a variable of ordinal type. While probably not very useful, it has always seemed to me to be a bit inconsistent that you can't. It might be useful in some kind of generated, generic, or otherwise parameterized code. If you intend these examples just to talk about language rules, OK. If examples of how to write actual code, they seem convoluted to me. They are better candidates for IF ladders. When the type of the expression is BOOLEAN, probably a single IF statement, with ELSE, is in order. The only situation I can think of where I would code something like this is if I wanted to force a compile-time error if the two expressions in the two alternatives ever became equal during maintenance, because someone changed one of the constants or types in a way that violated some algorithmic constraint. Or maybe just to emphasize that the case label expression should be constant and get the compiler to verify that. From hosking at cs.purdue.edu Sun Mar 14 19:28:56 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 14 Mar 2010 14:28:56 -0400 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: Correct. On 13 Mar 2010, at 22:52, Jay K wrote: > Hm. You mean, like I'm not supposed to be able to say: > > > PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= > BEGIN > CASE bool OF > | (a < b) => IO.Put("true\n"); > | (a > b) => IO.Put("false\n"); > END; > > > but I can say: > > > PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= > BEGIN > CASE bool OF > | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); > | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); > END; > > (I'd like to be able to say FIRST(a), etc., but I don't think it is allowed.) > > > > "constants" kind of seem like an optimization themselves. > Other than efficiency concerns, they could all be variables > initialized by module initializers, that the frontend doesn't > let you write to. Including de-optimizations like runtime > allocation/sizing of arrays based on const/non-const sizes. > > > Look at how I changed const to var already in m3core/unix, and then > had to change CASE to if/elseif ladders. > It's a minor matter of what the language lets you say, > among various basically equivalent forms. > The compiler could allow case to target non-constants. > It'd just mainly be slower. > There is the matter of CASE arms can't overlap, where if/elseif ladders can. > Because CASE is conceptually compared all at once where if/elseif > is sequential. > > > Ok. > > > > I see a few options at least. > > > m3middle could look basically like m3front, but with various checks like type checking removed > and assumed true. This would be an arduous task of duplication. > > M3CG.i3 Var and Target.Int could gain the following fields (or methods): > > > min_valid := FALSE; > max_valid := FALSE; > min := TInt.Zero; > max := TInt.Zero; > > > m3front could fill them in a few cases. > Initially none, but then a few as they are discovered to be easy, correct, efficient. > For constants, min = max value. > For subranges, min/max come the type. > The backends could use them (if valid) and possibly transform them. > e.g. MIN(a,b) with valid min/max yields an expression > where min is the min of the two mins, max is the min of the two maxes. > > > MIN([0..1],[2..3]) < 2 => TRUE. > > > Probably more correct would be: > > PROCEDURE declare_enum (xx: T; t: TypeUID; n_elts: INTEGER; s: BitSize) = > > PROCEDURE declare_subrange (xx: T; t, domain: TypeUID; > READONLY min, max: Target.Int; > s: BitSize) > > => already exist > > > and then > > > PROCEDURE load (xx: T; v: Var; o: ByteOffset; t: MType; u: ZType) = > PROCEDURE load_indirect (xx: T; o: ByteOffset; t: MType; u: ZType) = > PROCEDURE load_integer (xx: T; t: IType; READONLY i: Target.Int) = > PROCEDURE load_ordered (xx: T; t: MType; u: ZType; order: MemoryOrder) = > etc. > > > should all take an optional TypeUID. > > > The second proposal I've thought a bit more about and it seems very easy. > The last idea seems cleaner, but with possibly signifiant downside > in that e.g. Word.T is not a subrange. > > There's also some unclarity with respect to > Word.LT(expression, -1); > > -1 is a valid Word interpreted as a large number. > One would want to be careful not to break that. > I believe this is related to what you were saying, where operations > are typed, not values. > > > It could be that every operation needs min/max or typeuid (pairs > of them for binary operations). > > > Can we take a stab at something like this? > > > This might also serve to replace some of what m3back does with constants already, > since constants would fall out of subranges of length 1. > > > Thanks, > -Jay > > > From: hosking at cs.purdue.edu > Date: Sat, 13 Mar 2010 14:45:29 -0500 > To: hosking at cs.purdue.edu > CC: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: Re: [M3devel] comparisons vs. subranges > > Jay, > > I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 13 Mar 2010, at 14:23, Tony Hosking wrote: > > On 13 Mar 2010, at 13:26, Jay K wrote: > > Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. > > > What is the point of these functions Fold? > > So that constants can be manipulated as first-class values in the language. Seehttp://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. > > Why does it have these various "cost" computations? > > Which cost computations are you referring to? > > Granted, they are rough. > > > Should we beef up m3middle? > > That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. > > m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. > It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. > It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. > > Designing a decent optimising middle-end takes significant time and effort, and will require more thought. > > m3back is correct or extremely close to correct as far as I know. > It is missing atomic operations for 8, 16, 64 bit operands. > (PPC32 seems unable to support 64bit atomics btw.) > > Correct. > > I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. > > > I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Sat, 13 Mar 2010 13:03:18 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] comparisons vs. subranges > > Jay, > > I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 13 Mar 2010, at 05:19, Jay K wrote: > > <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; > <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; > > > Seems to me, the front end should notice these. > The first should always be TRUE. > And possibly, possibly warn. > The second should always be FALSE. > And possibly, possibly warn. > > "Generic" programming often hits this sort of thing though, a good reason not to warn. > Programmer might also be working with existing code and have changed INTEGER to CARDINAL. > Or be defending against future maintainers changing CARDINAL to INTEGER. > > > The backend isn't give enough information, because CARDINAL = INTEGER as far > as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. > The only way to get unsigned types is to use ADDRESS from what I see. > > > - Jay > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 19:30:59 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 18:30:59 +0000 Subject: [M3devel] atomics vs. locks.. Message-ID: http://msdn.microsoft.com/en-us/library/ee418650(VS.85).aspx MemoryBarrier was measured as taking 20-90 cycles. InterlockedIncrement was measured as taking 36-90 cycles. Acquiring or releasing a critical section was measured as taking 40-100 cycles. Acquiring or releasing a mutex was measured as taking about 750-2500 cycles perhaps just use critical sections... (I'm very ambivalent about atomics. It is just so darn difficult to do lockless programming. Granted, if you are a systems language, then you need to be able to imlement the locks themselves. Perhaps.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Sun Mar 14 22:20:26 2010 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sun, 14 Mar 2010 21:20:26 +0000 (GMT) Subject: [M3devel] bandwith Message-ID: <646699.86796.qm@web23603.mail.ird.yahoo.com> Hi Rodney: Well, I think the main issue as you point out is the architecture of the VM, as you know JVM is a Java based architecture so many of its structures are suited for that language and probably this is related to a compiler efficiency thing and thus you can say that a big language like Java needs an appropiate VM like JVM. I think that in M3 case what could be more suitable for real use is a SPIN powered vm like itself running in bare hardware with current hardware assisted virtualization such of current leading processors that would be a plus, it would lack supoort for more real machine languages (i.e C), but as it is running in kernel mode could give a better performance. In other architectures it could be run with kernel level virtualization and thus gaining better performance or similar even if its not a complete form of virtualization in bare hardware (by writing for SPIN M3 or other VM based language too and run in kernel mode like a M3 process, like the JVM M3 hosted if so, etc). I think having a kernel level virtual machine could be a gain in performance for some applications written for other machines or languages and or virtual machines or even the same kind of machines in M3 but in other systems (like running a DEC Unix or BSD flawor even with C libraries from inside SPIN). In fact SPIN was ported to NT as a driver for an intelligent NIC, thus allowed to run in bare hardware and at kernel level Thanks in advance, --- El dom, 14/3/10, Rodney M. Bates escribi?: > De: Rodney M. Bates > Asunto: Re: [M3devel] bandwith > Para: m3devel at elegosoft.com > Fecha: domingo, 14 de marzo, 2010 10:58 > > > Dirk Muysers wrote: > >? To make it work for all these different > platforms seems a herculean task and rather pointless > > endeavour. Why not use all that energy to try a > radically disruptive strategy such as a platform > > agnostic intermediate form targeting a JIT. > >? > > I have been a little way down this road at least three > times, with at > least two different languages, and it turns out not to be > what it would > appear at first glance, at least for the JVM. > > Java is a mostly object-oriented language, meaning the > non-object-oriented > types and operations are very limited.? Most of the > interesting stuff can > be done only with full-blown heap allocated objects and > dispatching methods. > Even the equivalents of records and arrays are this way inb > Java.? When you > don't need this full generality and power, you pay big > performance penalties > for heap allocation, and this is worse and much deeper than > the > performance penalties of an interpretive implementation. > (which, of > course, a JIT code generator partially avoids, and a > traditional > source-to-machine compiler avoids altogether, all with no > language > changes.) > > Many languages, Modula-3 included, give this generality for > when you need > it, but also give you lighter weight alternatives, for use > when appropriate. > > So when you try to implement some other language using JVM, > you soon > discover that JVM lacks a lot of operations needed for the > not-so- > impoverished non-object-oriented types of the language. > > Whether or how badly the .net/mono virtual machine suffers > from this, I > don't know.? It apparently was designed to support a > much wider range > of languages, but I seem to recall hearing some discussions > that suggested > it had some of the same limitations. > > Of course, we could invent our own virtual machine (M-code? > MVM?) ;-). > > This whole approach is of course, not very suitable for > systems programming, > one of Modula-3's strengths. > > > > > d. muysers > From jay.krell at cornell.edu Mon Mar 15 03:51:54 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 15 Mar 2010 02:51:54 +0000 Subject: [M3devel] Mx86.m3 vs. stackx86.m3? Message-ID: Does anyone understand and can explain the rationale to put some things in Mx86.m3 and some in Stackx86.m3? It seems arbitrary in many places to me. For exampe, in the release branch, compare M3x86 shift, shift_left, shift_right. shift_left and shift_right do the work themselves, but shift calls stack.doshift. In this I case merged all the code into stack.doshift, with an enum to indicate which of the three types of shifts to do. rotate is still the old way. (I don't yet inline 64bit rotates without constants.) Also, I assume "vstack" means "virtual stack"? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 16 01:54:41 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 00:54:41 +0000 Subject: [M3devel] arbitrary use of LONGINT for testing? target/ABI alteration/generation via command line? Message-ID: Tony et. al. I'm wondering/fishing if you any ideas, like a cm3 command line switch, that might be used to make INTEGER 64bits to increase coverage/testing of 64bit integer support. The problem is *at least* interfacing with external code/files, if not breaking everything. Maybe also a problem around sizeof(INTEGER) vs. sizeof(ADDRESS). Probably such a setting would grow ADDRESS too. Like, maybe stuff like m3core/unix, m3core/win32 should never use plain INTEGER or LONGINT, but "BITS FOR", to allow this to work? Or maybe <* EXTERNAL *> would be the hint? No, <* EXTERNAL *> isn't adequate, due to the case of callbacks. I'd like to somehow, without too much effort, significantly increase testing of LONGINT. Maybe a wierdo platform NT386_64 that does this just for test purposes? I might try that, if "BITS FOR" is enough to actually let it work (interface with C). (or generally, cm3 -test64 would append "_TEST64" to target name, so we could have LINUXLIBC6_TEST64, I386_DARWIN_TEST64, etc.) I can do some initial experiments along this -test64 line, see if it completely blows up or not. A few minutes thought suggests it should work easily and provide value. (There's probably something similar to try around a 16 bit CHAR. vs. BITS 8 FOR CHAR, but I'm not going there..) Such command line driven target/ABI alterations seem like a somewhat good idea. e.g. cm3 -userthreads would append _USERTHREADS. cm3 -extended80 would use an 80 or 96 bit extended on x86 (96 bits for alignment). -extended128 on ppc etc. cm3 -msvc80 could probe around in enviroment for a Visual C++ 8.0 compiler/linker/header/libraries and append "_MSVC80" to target, else fail. I'll just try -test64 for now. The rest aren't as immediately useful. -userthreads would be my next "favorate". Hm. Instead of -test64, how about -64. If integer is 32 bits, change it to 64 and append _64 (and address). If integer is already 64bits, do nothing. This is quite different than gcc's -m64 though. Maybe not good that way. You can see parallels in several older C compilers. Metrowerks for Mac/68K let you chose integer size and either double or extended size (I forget which). That gave you a cross product set of ABIs, each with their own libraries. Similarly you could chose to issue FPU instructions directly, which might have altered the ABI, or maybe the alternate libraries were just faster. Similarly Microsoft and memory models. Various switches changed the size of a pointer and implied a need for different libraries. Current gcc and ppc floating point sizes I believe is a contemporary example. Even sometimes the affect that -pthread has -- chosing different libraries (unnecessary mess imho, but one that seems to persist on HP-UX.) It's kind of an ugly area, to offer too many choices, produces confusion, but the limited important goal of increasing 64bit integer testing seems worth going here. And increasing userthread testing. People have expressed an interest in higher precision floating point, esp. 80 bit on x86, so this maybe a good way to proceed there also. You could also imagine flags -SSE, -SSE2, etc. It is important to only support a very limited number of these flags. Large cross products are not fun to worry about and test. But for example, with SSE, that nets you several targets all at once, without per-target work. (I lose the term "SSE" loosely. Could be I mean "SSE2" or 3 -- whatever is sufficient to largely replace the stack based x87.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 02:00:27 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 15 Mar 2010 20:00:27 -0500 Subject: [M3devel] arbitrary use of LONGINT for testing? target/ABI alteration/generation via command line? In-Reply-To: References: Message-ID: <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu> Let's not right now. We need to get the release out. I suggest a freeze on compiler changes for now. Bug fixes only! Sent from my iPhone On Mar 15, 2010, at 7:54 PM, Jay K wrote: > Tony et. al. I'm wondering/fishing if you any ideas, like a cm3 > command line switch, that might be used to make INTEGER 64bits to > increase coverage/testing of 64bit integer support. > > > The problem is *at least* interfacing with external code/files, if > not breaking everything. > Maybe also a problem around sizeof(INTEGER) vs. sizeof(ADDRESS). > Probably such a setting would grow ADDRESS too. > > > Like, maybe stuff like m3core/unix, m3core/win32 should never use > plain INTEGER or LONGINT, but "BITS FOR", to allow this to work? Or > maybe <* EXTERNAL *> would be the hint? > No, <* EXTERNAL *> isn't adequate, due to the case of callbacks. > > > I'd like to somehow, without too much effort, significantly increase > testing of LONGINT. > > > Maybe a wierdo platform NT386_64 that does this just for test > purposes? > I might try that, if "BITS FOR" is enough to actually let it work > (interface with C). > (or generally, cm3 -test64 would append "_TEST64" to target name, > so we could have LINUXLIBC6_TEST64, I386_DARWIN_TEST64, etc.) > > > I can do some initial experiments along this -test64 line, see if it > completely blows up or not. > A few minutes thought suggests it should work easily and provide > value. > > > (There's probably something similar to try around a 16 bit CHAR. vs. > BITS 8 FOR CHAR, but I'm not going there..) > > > Such command line driven target/ABI alterations seem like a somewhat > good idea. > e.g. cm3 -userthreads would append _USERTHREADS. > cm3 -extended80 would use an 80 or 96 bit extended on x86 (96 bits > for alignment). > -extended128 on ppc > etc. > > > cm3 -msvc80 could probe around in enviroment for a Visual C++ 8.0 > compiler/linker/header/libraries and append "_MSVC80" to target, > else fail. > > > I'll just try -test64 for now. > The rest aren't as immediately useful. > -userthreads would be my next "favorate". > > Hm. Instead of -test64, how about -64. > If integer is 32 bits, change it to 64 and append _64 (and address). > If integer is already 64bits, do nothing. > > > This is quite different than gcc's -m64 though. > Maybe not good that way. > > > You can see parallels in several older C compilers. > > Metrowerks for Mac/68K let you chose integer size and either double > or extended size (I forget which). > That gave you a cross product set of ABIs, each with their own > libraries. > > > Similarly you could chose to issue FPU instructions directly, which > might have altered the ABI, or maybe the alternate libraries were > just faster. > > > Similarly Microsoft and memory models. Various switches changed the > size of a pointer and implied a need for different libraries. > > > Current gcc and ppc floating point sizes I believe is a contemporary > example. > Even sometimes the affect that -pthread has -- chosing different > libraries (unnecessary > mess imho, but one that seems to persist on HP-UX.) > > > It's kind of an ugly area, to offer too many choices, produces > confusion, but the limited important goal of increasing 64bit > integer testing seems worth going here. And increasing userthread > testing. People have expressed an interest in higher precision > floating point, esp. 80 bit on x86, so this maybe a good way to > proceed there also. > You could also imagine flags -SSE, -SSE2, etc. > > > It is important to only support a very limited number of these flags. > Large cross products are not fun to worry about and test. > But for example, with SSE, that nets you several targets all at > once, without per-target work. > > (I lose the term "SSE" loosely. Could be I mean "SSE2" or 3 -- > whatever is sufficient to largely replace the stack based x87.) > > > - Jay From jay.krell at cornell.edu Tue Mar 16 02:41:37 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 01:41:37 +0000 Subject: [M3devel] release status [was something else] In-Reply-To: <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu> References: , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu> Message-ID: Release status as far as I know: - cvsupd hang Reported over a year ago. Needs investigation. Can anyone confirm it with a recent version? And, as Tony asked, with user threads? - NT386 has no 64bit longint in release branch. Partly the point of this question. - Should maybe improve build/packaging to get NT386-VC80.msi, NT386-VC90.msi etc. (not the same as "target/abi alteration via command line" expressed below, though that is a possibility, but separate clean builds for now and/or separate CVS checkouts); really more of a Hudson task now, to have multiple build environments installed and used, I think I already updated the .msi building to append the tag. Olaf? - I believe there is "new" divergence in m3front between head and release -- whatever came along with the TInt changes. Otherwise I'd have to look through trac. There's maybe some stuff not merged from release to head, but that's ok for release. (I need to check which branch Randy fixed the examples in.) I need to re-test the .msis. - Jay > From: hosking at cs.purdue.edu > To: jay.krell at cornell.edu > Subject: Re: [M3devel] arbitrary use of LONGINT for testing? target/ABI alteration/generation via command line? > Date: Mon, 15 Mar 2010 20:00:27 -0500 > CC: m3devel at elegosoft.com > > Let's not right now. We need to get the release out. I suggest a > freeze on compiler changes for now. Bug fixes only! > > Sent from my iPhone > > On Mar 15, 2010, at 7:54 PM, Jay K wrote: > > > Tony et. al. I'm wondering/fishing if you any ideas, like a cm3 > > command line switch, that might be used to make INTEGER 64bits to > > increase coverage/testing of 64bit integer support. > > > > > > The problem is *at least* interfacing with external code/files, if > > not breaking everything. > > Maybe also a problem around sizeof(INTEGER) vs. sizeof(ADDRESS). > > Probably such a setting would grow ADDRESS too. > > > > > > Like, maybe stuff like m3core/unix, m3core/win32 should never use > > plain INTEGER or LONGINT, but "BITS FOR", to allow this to work? Or > > maybe <* EXTERNAL *> would be the hint? > > No, <* EXTERNAL *> isn't adequate, due to the case of callbacks. > > > > > > I'd like to somehow, without too much effort, significantly increase > > testing of LONGINT. > > > > > > Maybe a wierdo platform NT386_64 that does this just for test > > purposes? > > I might try that, if "BITS FOR" is enough to actually let it work > > (interface with C). > > (or generally, cm3 -test64 would append "_TEST64" to target name, > > so we could have LINUXLIBC6_TEST64, I386_DARWIN_TEST64, etc.) > > > > > > I can do some initial experiments along this -test64 line, see if it > > completely blows up or not. > > A few minutes thought suggests it should work easily and provide > > value. > > > > > > (There's probably something similar to try around a 16 bit CHAR. vs. > > BITS 8 FOR CHAR, but I'm not going there..) > > > > > > Such command line driven target/ABI alterations seem like a somewhat > > good idea. > > e.g. cm3 -userthreads would append _USERTHREADS. > > cm3 -extended80 would use an 80 or 96 bit extended on x86 (96 bits > > for alignment). > > -extended128 on ppc > > etc. > > > > > > cm3 -msvc80 could probe around in enviroment for a Visual C++ 8.0 > > compiler/linker/header/libraries and append "_MSVC80" to target, > > else fail. > > > > > > I'll just try -test64 for now. > > The rest aren't as immediately useful. > > -userthreads would be my next "favorate". > > > > Hm. Instead of -test64, how about -64. > > If integer is 32 bits, change it to 64 and append _64 (and address). > > If integer is already 64bits, do nothing. > > > > > > This is quite different than gcc's -m64 though. > > Maybe not good that way. > > > > > > You can see parallels in several older C compilers. > > > > Metrowerks for Mac/68K let you chose integer size and either double > > or extended size (I forget which). > > That gave you a cross product set of ABIs, each with their own > > libraries. > > > > > > Similarly you could chose to issue FPU instructions directly, which > > might have altered the ABI, or maybe the alternate libraries were > > just faster. > > > > > > Similarly Microsoft and memory models. Various switches changed the > > size of a pointer and implied a need for different libraries. > > > > > > Current gcc and ppc floating point sizes I believe is a contemporary > > example. > > Even sometimes the affect that -pthread has -- chosing different > > libraries (unnecessary > > mess imho, but one that seems to persist on HP-UX.) > > > > > > It's kind of an ugly area, to offer too many choices, produces > > confusion, but the limited important goal of increasing 64bit > > integer testing seems worth going here. And increasing userthread > > testing. People have expressed an interest in higher precision > > floating point, esp. 80 bit on x86, so this maybe a good way to > > proceed there also. > > You could also imagine flags -SSE, -SSE2, etc. > > > > > > It is important to only support a very limited number of these flags. > > Large cross products are not fun to worry about and test. > > But for example, with SSE, that nets you several targets all at > > once, without per-target work. > > > > (I lose the term "SSE" loosely. Could be I mean "SSE2" or 3 -- > > whatever is sufficient to largely replace the stack based x87.) > > > > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 16 11:39:09 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 11:39:09 +0100 Subject: [M3devel] release status [was something else] In-Reply-To: References: , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu> Message-ID: <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Quoting Jay K : > Release status as far as I know: > > - cvsupd hang > Reported over a year ago. Needs investigation. > Can anyone confirm it with a recent version? > And, as Tony asked, with user threads? If we can reproduce this, we should fix it; if not, I'd say this isn't relevant for the release. We can try to use a cvsupd from the release branch on birch serving the cm3 repository on a different port. Let's see if that works. Maybe tonight. > - NT386 has no 64bit longint in release branch. Partly the point of > this question. I thought Randy was going to do some more testing. If the m3back LONGINT changes don't break anything else, I'd be for just merging them to the branch and see if the builds and tests succeed. Of course testing with some other real applications would be great, but we must not delay endlessly for that. > - Should maybe improve build/packaging to get NT386-VC80.msi, > NT386-VC90.msi etc. (not the same as "target/abi alteration via > command line" expressed below, though that is a possibility, but > separate clean builds for now and/or separate CVS checkouts); really > more of a Hudson task now, to have multiple build environments > installed and used, I think I already updated the .msi building to > append the tag. Olaf? I'd rather not touch the NT386 setup on our virtual machine; perhaps you could provide another machine to run the Hudson jobs with a different Microsoft tools setup? > - I believe there is "new" divergence in m3front between head and > release -- whatever came along with the TInt changes. Is this relevant for the release? Or can we just ignore it? > Otherwise I'd have to look through trac. That's always a good idea. > There's maybe some stuff not merged from release to head, but that's > ok for release. > > (I need to check which branch Randy fixed the examples in.) > > I need to re-test the .msis. We still need to add them to the WWW export/display, don't we? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 16 12:21:36 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 11:21:36 +0000 Subject: [M3devel] release status [was something else] In-Reply-To: <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: Responding to just parts: > > - NT386 has no 64bit longint in release branch. Partly the point of > > this question. > > I thought Randy was going to do some more testing. If the m3back LONGINT > changes don't break anything else, I'd be for just merging them to the It is a big change. It is dependent on TInt/TWord changes, though that is flexible. I could copy and rename them "M3BackInt". Or we could take them -- they go with m3front changes. Or we could even do a little experiment: The reason I use TInt/TWord is to get 64bit integers portably to 32bit hosts. Previously INTEGER/Word was used. I could try using LONGINT/Long instead. However it is still a large change. There are some other unrelated changes, nothing too significant on its own (I always say), but it adds up: It isn't messed up, but it is maybe "beyond recognition" compared to the previous. (as was recently suggested) some renaming for, I claim, clarity some small optimizations (e.g. using shorter encodings sometimes, like a byte instead of a dword for constants in instructions, or equivalent shorter constructs like or reg,-1 instead of mov reg,-1), atomics support, actually in very good shape now, but could use more testing and optimization "rewrite" of insert/extract to no longer use the tables, I've always been bothered by those tables "rewrite" of set_singleton/set_member to be *much* smaller/faster set_singleton is just the bts instruction, instead of a function call set_member is just bt instruction plus carry flag extraction, instead of a function call cleanup of the way epilogues are sometimes "patched in" Only save/restore non-volatile registers that are actually used. Probably other stuff I'm forgetting. There's also small related changes in m3objfile: support for outputing more than 4 bytes at a time, support for "backing up". Some extra commenting. Some formating consistency (then always at end of line, else always on its own line). "Namespace flattening" (from foo import a,b,c). I need to test 64bit subrange checking. The code is a little fishy in that it deals with single registers. It is an optimization and could probably just be pessimized. Or maybe easy to fix. There's also the question as to if the TInt use is all subtely wrong as Tony seems to say. I don't understand yet. It *seems* to work well and it didn't previously pay much attention to types, it just used INTEGER everywhere. The changes are overall large. Diff the two trees, just in m3back. I actually think my earlier question might be the way to go to dramatically increase testing/coverage/confidence -- cm3 -test64 appends "_TEST64" to the build dir name (maybe maybe not the target name) and sets WORD_SIZE = "64" and BITSIZE(INTEGER) and ADDRESS to 64. I'd even consider something "simpler" where I actually create another complete target, I386_NT_TEST64, including the various entries in m3makefiles, Target.i3, etc. Maybe I can just try this locally with a one line change in Target.m3 though. I'll try that first. > branch and see if the builds and tests succeed. Of course testing with > some other real applications would be great, but we must not delay > endlessly for that. I have to accept releasing with 32bit LONGINT on NT386, due to the large overall change. Hopefully we arrange for more frequent releases somehow. > > - Should maybe improve build/packaging to get NT386-VC80.msi, > > I'd rather not touch the NT386 setup on our virtual machine; I probably won't leave a machine running 24/7. Can I just run the automation manually? Recall Cygwin sshd didn't work adequately at the time. Well, yes, I can always run whatever automation. Somewhat it is a matter of principle of going through the more official more automated process vs. a less official, more error prone, less trusted manual process. Installing additional toolsets on the VM really should work ok, without breaking the existing. Can you try? Maybe this: You create one .msi using the existing process. We'll make sure it is stamped "-VC90" or whatnot. I'll build a whole bunch of others, and they be available as "alternates", and we'll exclude the one corresponding to yours? You provide e.g. cm3-5.8-I386_NT-VC80.msi and I'll provide cm3-5.8-I386_NT-VC20.msi cm3-5.8-I386_NT-VC40.msi cm3-5.8-I386_NT-VC41.msi cm3-5.8-I386_NT-VC42.msi cm3-5.8-I386_NT-VC50.msi cm3-5.8-I386_NT-VC60.msi cm3-5.8-I386_NT-VC70.msi cm3-5.8-I386_NT-VC71.msi cm3-5.8-I386_NT-VC90.msi I realize some of these are quite old and out of use, but it's pretty simple to produce them all. > > - I believe there is "new" divergence in m3front between head and > Is this relevant for the release? Or can we just ignore it? I'd have to look, or Tony should say. To some extent it is tied with if we take NT386 64bit longint. > msi > We still need to add them to the WWW export/display, don't we? Definitely. Sorry of this isn't edited well..took a while already... - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 16 13:32:49 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 12:32:49 +0000 Subject: [M3devel] cvsup Message-ID: I can reproduce the cvsup problem. An important point to consider is that cvsupd forks a server? Maybe that messes up initialization? I'll dig further. I think these steps are complete. I'll test them on a second clean system. You don't need any real data. jay at xlin2:~$ cat cvsupd.cfg *default tag=. *default host=xlin2. *default prefix=/home/jay *default base=/home/jay/var/db *default release=cvs delete use-rel-suffix compress src-all mkdir -p $HOME/var/db mkdir -p $HOME/data/cvsupd pkill cvsupd # cleanup any previous You don't really need any data. cvsup/cvsupd will issue an error in the "working" case, and hang in the non-working case. run the server: jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd 2010.03.16 05:24:25 PDT [22555]: CVSup server started 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 2010.03.16 05:24:25 PDT [22555]: Ready to service requests run the client: /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg Parsing supfile "/home/jay/cvsupd.cfg" Connecting to xlin2. Connected to xlin2. Server software version: 2010-03-05 Negotiating file attribute support Exchanging collection information Server message: Unknown collection "src-all" Establishing multiplexed-mode data connection Running Skipping collection src-all/cvs Shutting down connection to server Finished successfully it is able to talk to the server, and then fails. Ok -- at least it talked to the server. The server then exits too. I think that is correct (read the usage on the -C option). Then try with -C. The bug says -C 2. Let's use -C 1. They behave the same for our purposes. Just add -C 1 to the above server. Run the same client. The client hangs -- it never connects to the server. I reproduced this on LINUXLIBC6 and PPC_LINUX. Maybe more soon. I'm just rebuilding first. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 16 13:38:30 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 13:38:30 +0100 Subject: [M3devel] release status [was something else] In-Reply-To: References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: <20100316133830.8pcec289kcsc0sgk@mail.elegosoft.com> Quoting Jay K : > It is a big change. [...] > I actually think my earlier question might be the way to go to dramatically > increase testing/coverage/confidence -- cm3 -test64 appends "_TEST64" > to the build dir name (maybe maybe not the target name) and sets > WORD_SIZE = "64" and BITSIZE(INTEGER) and ADDRESS to 64. > > I'd even consider something "simpler" where I actually create > another complete > target, I386_NT_TEST64, including the various entries in > m3makefiles, Target.i3, etc. > > Maybe I can just try this locally with a one line change in Target.m3 though. > > I'll try that first. That would be good. [...] > I have to accept releasing with 32bit LONGINT on NT386, due to the large > overall change. > > Hopefully we arrange for more frequent releases somehow. So you think we should not risk the merge and release without the m3back changes now? >> > - Should maybe improve build/packaging to get NT386-VC80.msi, >> >> I'd rather not touch the NT386 setup on our virtual machine; > > I probably won't leave a machine running 24/7. > > Can I just run the automation manually? You can copy and run all the Hudson jobs manually. I'd suggest a parametric setup for the differences though. > Recall Cygwin sshd didn't work adequately at the time. > Well, yes, I can always run whatever automation. Somewhat > it is a matter of principle of going through the more official > more automated > process vs. a less official, more error prone, less trusted > manual process. > Installing additional toolsets on the VM really should work ok, > without breaking > the existing. Can you try? I cannot really do that without GUI access; and I'm afraid there are currently no ressources to setup all those tools (Kay's busy with other work). > Maybe this: You create one .msi using the existing process. We'll make > sure it is stamped "-VC90" or whatnot. > > I'll build a whole bunch of others, and they be available as "alternates", > and we'll exclude the one corresponding to yours? > > You provide e.g. cm3-5.8-I386_NT-VC80.msi and I'll provide > > cm3-5.8-I386_NT-VC20.msi > cm3-5.8-I386_NT-VC40.msi > cm3-5.8-I386_NT-VC41.msi > cm3-5.8-I386_NT-VC42.msi > cm3-5.8-I386_NT-VC50.msi > cm3-5.8-I386_NT-VC60.msi > cm3-5.8-I386_NT-VC70.msi > cm3-5.8-I386_NT-VC71.msi > > cm3-5.8-I386_NT-VC90.msi Do we really need all those? We would not only need the msi installers, but the other packages would have to be compiled and linked with the different tools, too, or they'd be incompatible, wouldn't they? > I realize some of these are quite old and out of use, but it's > pretty simple to produce them all. Of course you can contribute whatever installation archives you can create. I'd rather have a defined set of target platforms and build procedures for the official release though. So I think we should limit our set of supported tool sets. What would you suggest that is most likely to be really useful? > > > - I believe there is "new" divergence in m3front between head and > > Is this relevant for the release? Or can we just ignore it? > > I'd have to look, or Tony should say. > > To some extent it is tied with if we take NT386 64bit longint. > > We still need to add them to the WWW export/display, don't we? > > Definitely. I'll do that. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From hosking at cs.purdue.edu Tue Mar 16 16:07:47 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 11:07:47 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: References: , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu> Message-ID: On 15 Mar 2010, at 21:41, Jay K wrote: > Release status as far as I know: > > - cvsupd hang > Reported over a year ago. Needs investigation. > Can anyone confirm it with a recent version? > And, as Tony asked, with user threads? If anyone gets a snapshot please also make sure to get a backtrace for all the threads. > - NT386 has no 64bit longint in release branch. Partly the point of this question. > > > - Should maybe improve build/packaging to get NT386-VC80.msi, NT386-VC90.msi etc. (not the same as "target/abi alteration via command line" expressed below, though that is a possibility, but separate clean builds for now and/or separate CVS checkouts); really more of a Hudson task now, to have multiple build environments installed and used, I think I already updated the .msi building to append the tag. Olaf? > > > - I believe there is "new" divergence in m3front between head and release -- whatever came along with the TInt changes. Those changes will need bringing over from head to release if Jay's m3back 64-bit support for NT386 are also brought over. I believe there are a few dependencies. > Otherwise I'd have to look through trac. > > > There's maybe some stuff not merged from release to head, but that's ok for release. > (I need to check which branch Randy fixed the examples in.) > I need to re-test the .msis. > > > - Jay > > > > From: hosking at cs.purdue.edu > > To: jay.krell at cornell.edu > > Subject: Re: [M3devel] arbitrary use of LONGINT for testing? target/ABI alteration/generation via command line? > > Date: Mon, 15 Mar 2010 20:00:27 -0500 > > CC: m3devel at elegosoft.com > > > > Let's not right now. We need to get the release out. I suggest a > > freeze on compiler changes for now. Bug fixes only! > > > > Sent from my iPhone > > > > On Mar 15, 2010, at 7:54 PM, Jay K wrote: > > > > > Tony et. al. I'm wondering/fishing if you any ideas, like a cm3 > > > command line switch, that might be used to make INTEGER 64bits to > > > increase coverage/testing of 64bit integer support. > > > > > > > > > The problem is *at least* interfacing with external code/files, if > > > not breaking everything. > > > Maybe also a problem around sizeof(INTEGER) vs. sizeof(ADDRESS). > > > Probably such a setting would grow ADDRESS too. > > > > > > > > > Like, maybe stuff like m3core/unix, m3core/win32 should never use > > > plain INTEGER or LONGINT, but "BITS FOR", to allow this to work? Or > > > maybe <* EXTERNAL *> would be the hint? > > > No, <* EXTERNAL *> isn't adequate, due to the case of callbacks. > > > > > > > > > I'd like to somehow, without too much effort, significantly increase > > > testing of LONGINT. > > > > > > > > > Maybe a wierdo platform NT386_64 that does this just for test > > > purposes? > > > I might try that, if "BITS FOR" is enough to actually let it work > > > (interface with C). > > > (or generally, cm3 -test64 would append "_TEST64" to target name, > > > so we could have LINUXLIBC6_TEST64, I386_DARWIN_TEST64, etc.) > > > > > > > > > I can do some initial experiments along this -test64 line, see if it > > > completely blows up or not. > > > A few minutes thought suggests it should work easily and provide > > > value. > > > > > > > > > (There's probably something similar to try around a 16 bit CHAR. vs. > > > BITS 8 FOR CHAR, but I'm not going there..) > > > > > > > > > Such command line driven target/ABI alterations seem like a somewhat > > > good idea. > > > e.g. cm3 -userthreads would append _USERTHREADS. > > > cm3 -extended80 would use an 80 or 96 bit extended on x86 (96 bits > > > for alignment). > > > -extended128 on ppc > > > etc. > > > > > > > > > cm3 -msvc80 could probe around in enviroment for a Visual C++ 8.0 > > > compiler/linker/header/libraries and append "_MSVC80" to target, > > > else fail. > > > > > > > > > I'll just try -test64 for now. > > > The rest aren't as immediately useful. > > > -userthreads would be my next "favorate". > > > > > > Hm. Instead of -test64, how about -64. > > > If integer is 32 bits, change it to 64 and append _64 (and address). > > > If integer is already 64bits, do nothing. > > > > > > > > > This is quite different than gcc's -m64 though. > > > Maybe not good that way. > > > > > > > > > You can see parallels in several older C compilers. > > > > > > Metrowerks for Mac/68K let you chose integer size and either double > > > or extended size (I forget which). > > > That gave you a cross product set of ABIs, each with their own > > > libraries. > > > > > > > > > Similarly you could chose to issue FPU instructions directly, which > > > might have altered the ABI, or maybe the alternate libraries were > > > just faster. > > > > > > > > > Similarly Microsoft and memory models. Various switches changed the > > > size of a pointer and implied a need for different libraries. > > > > > > > > > Current gcc and ppc floating point sizes I believe is a contemporary > > > example. > > > Even sometimes the affect that -pthread has -- chosing different > > > libraries (unnecessary > > > mess imho, but one that seems to persist on HP-UX.) > > > > > > > > > It's kind of an ugly area, to offer too many choices, produces > > > confusion, but the limited important goal of increasing 64bit > > > integer testing seems worth going here. And increasing userthread > > > testing. People have expressed an interest in higher precision > > > floating point, esp. 80 bit on x86, so this maybe a good way to > > > proceed there also. > > > You could also imagine flags -SSE, -SSE2, etc. > > > > > > > > > It is important to only support a very limited number of these flags. > > > Large cross products are not fun to worry about and test. > > > But for example, with SSE, that nets you several targets all at > > > once, without per-target work. > > > > > > (I lose the term "SSE" loosely. Could be I mean "SSE2" or 3 -- > > > whatever is sufficient to largely replace the stack based x87.) > > > > > > > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 16:21:41 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 11:21:41 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: Ah, yes, one issue about bringing over m3front changes is that it also includes the atomics support. I don't think we want to do this in this release. So, this argues that we hold off on releasing the NT386 64-bit LONGINT support for now. Thoughts? On 16 Mar 2010, at 07:21, Jay K wrote: > Responding to just parts: > > > > > - NT386 has no 64bit longint in release branch. Partly the point of > > > this question. > > > > I thought Randy was going to do some more testing. If the m3back LONGINT > > changes don't break anything else, I'd be for just merging them to the > > > It is a big change. > It is dependent on TInt/TWord changes, though that is flexible. > I could copy and rename them "M3BackInt". > Or we could take them -- they go with m3front changes. > > > Or we could even do a little experiment: The reason I use TInt/TWord > is to get 64bit integers portably to 32bit hosts. Previously INTEGER/Word was used. > I could try using LONGINT/Long instead. > > > However it is still a large change. There are some other unrelated changes, nothing > too significant on its own (I always say), but it adds up: > It isn't messed up, but it is maybe "beyond recognition" compared to the previous. > (as was recently suggested) > > some renaming for, I claim, clarity > > some small optimizations (e.g. using shorter encodings sometimes, like > a byte instead of a dword for constants in instructions, or > equivalent shorter constructs like or reg,-1 instead of mov reg,-1), > > atomics support, actually in very good shape now, but could use more testing and optimization > > "rewrite" of insert/extract to no longer use the tables, I've always > been bothered by those tables > > "rewrite" of set_singleton/set_member to be *much* smaller/faster > set_singleton is just the bts instruction, instead of a function call > set_member is just bt instruction plus carry flag extraction, instead of a function call > > cleanup of the way epilogues are sometimes "patched in" > > Only save/restore non-volatile registers that are actually used. > > Probably other stuff I'm forgetting. > > There's also small related changes in m3objfile: support for outputing more than 4 > bytes at a time, support for "backing up". > Some extra commenting. > Some formating consistency (then always at end of line, else always on its own line). > "Namespace flattening" (from foo import a,b,c). > > I need to test 64bit subrange checking. > The code is a little fishy in that it deals with single registers. > It is an optimization and could probably just be pessimized. > Or maybe easy to fix. > > > There's also the question as to if the TInt use is all subtely wrong as Tony seems to say. > I don't understand yet. > It *seems* to work well and it didn't previously pay much attention to types, it > just used INTEGER everywhere. > > > The changes are overall large. > Diff the two trees, just in m3back. > > > I actually think my earlier question might be the way to go to dramatically > increase testing/coverage/confidence -- cm3 -test64 appends "_TEST64" > to the build dir name (maybe maybe not the target name) and sets > WORD_SIZE = "64" and BITSIZE(INTEGER) and ADDRESS to 64. > I'd even consider something "simpler" where I actually create another complete > target, I386_NT_TEST64, including the various entries in m3makefiles, Target.i3, etc. > > > Maybe I can just try this locally with a one line change in Target.m3 though. > I'll try that first. > > > > branch and see if the builds and tests succeed. Of course testing with > > some other real applications would be great, but we must not delay > > endlessly for that. > > > I have to accept releasing with 32bit LONGINT on NT386, due to the large > overall change. > Hopefully we arrange for more frequent releases somehow. > > > > > > - Should maybe improve build/packaging to get NT386-VC80.msi, > > > > I'd rather not touch the NT386 setup on our virtual machine; > > > I probably won't leave a machine running 24/7. > Can I just run the automation manually? > Recall Cygwin sshd didn't work adequately at the time. > Well, yes, I can always run whatever automation. Somewhat > it is a matter of principle of going through the more official more automated > process vs. a less official, more error prone, less trusted manual process. > > > Installing additional toolsets on the VM really should work ok, without breaking > the existing. Can you try? > > > Maybe this: You create one .msi using the existing process. We'll make > sure it is stamped "-VC90" or whatnot. > I'll build a whole bunch of others, and they be available as "alternates", > and we'll exclude the one corresponding to yours? > You provide e.g. cm3-5.8-I386_NT-VC80.msi and I'll provide > cm3-5.8-I386_NT-VC20.msi > cm3-5.8-I386_NT-VC40.msi > cm3-5.8-I386_NT-VC41.msi > cm3-5.8-I386_NT-VC42.msi > cm3-5.8-I386_NT-VC50.msi > cm3-5.8-I386_NT-VC60.msi > cm3-5.8-I386_NT-VC70.msi > cm3-5.8-I386_NT-VC71.msi > > cm3-5.8-I386_NT-VC90.msi > > > I realize some of these are quite old and out of use, but it's pretty simple to produce > them all. > > > > > - I believe there is "new" divergence in m3front between head and > > Is this relevant for the release? Or can we just ignore it? > > I'd have to look, or Tony should say. > To some extent it is tied with if we take NT386 64bit longint. > > > > msi > > We still need to add them to the WWW export/display, don't we? > > > Definitely. > > > Sorry of this isn't edited well..took a while already... > > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 16:23:34 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 11:23:34 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: <20100316133830.8pcec289kcsc0sgk@mail.elegosoft.com> References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> <20100316133830.8pcec289kcsc0sgk@mail.elegosoft.com> Message-ID: <720A9819-E452-4CEB-A0AA-149B1FF6D88E@cs.purdue.edu> On 16 Mar 2010, at 08:38, Olaf Wagner wrote: > Quoting Jay K : > >> It is a big change. > [...] >> I actually think my earlier question might be the way to go to dramatically >> increase testing/coverage/confidence -- cm3 -test64 appends "_TEST64" >> to the build dir name (maybe maybe not the target name) and sets >> WORD_SIZE = "64" and BITSIZE(INTEGER) and ADDRESS to 64. >> >> I'd even consider something "simpler" where I actually create another complete >> target, I386_NT_TEST64, including the various entries in m3makefiles, Target.i3, etc. >> >> Maybe I can just try this locally with a one line change in Target.m3 though. >> >> I'll try that first. > That would be good. That has pervasive implications. I don't know that there are not large swathes of code that assume that BITSIZE(INTEGER)=BITSIZE(ADDRESS). > > [...] >> I have to accept releasing with 32bit LONGINT on NT386, due to the large >> overall change. Yes, I think that is the way we need to go -- defer 64-bit NT386 LONGINT for now. >> >> Hopefully we arrange for more frequent releases somehow. > > So you think we should not risk the merge and release without the > m3back changes now? Yes. > >>> > - Should maybe improve build/packaging to get NT386-VC80.msi, >>> >>> I'd rather not touch the NT386 setup on our virtual machine; >> >> I probably won't leave a machine running 24/7. >> >> Can I just run the automation manually? > > You can copy and run all the Hudson jobs manually. I'd suggest a > parametric setup for the differences though. > >> Recall Cygwin sshd didn't work adequately at the time. >> Well, yes, I can always run whatever automation. Somewhat >> it is a matter of principle of going through the more official more automated >> process vs. a less official, more error prone, less trusted manual process. > >> Installing additional toolsets on the VM really should work ok, without breaking >> the existing. Can you try? > > I cannot really do that without GUI access; and I'm afraid there are > currently no ressources to setup all those tools (Kay's busy with other > work). > >> Maybe this: You create one .msi using the existing process. We'll make >> sure it is stamped "-VC90" or whatnot. >> >> I'll build a whole bunch of others, and they be available as "alternates", >> and we'll exclude the one corresponding to yours? >> >> You provide e.g. cm3-5.8-I386_NT-VC80.msi and I'll provide >> >> cm3-5.8-I386_NT-VC20.msi >> cm3-5.8-I386_NT-VC40.msi >> cm3-5.8-I386_NT-VC41.msi >> cm3-5.8-I386_NT-VC42.msi >> cm3-5.8-I386_NT-VC50.msi >> cm3-5.8-I386_NT-VC60.msi >> cm3-5.8-I386_NT-VC70.msi >> cm3-5.8-I386_NT-VC71.msi >> >> cm3-5.8-I386_NT-VC90.msi > > Do we really need all those? We would not only need the msi installers, > but the other packages would have to be compiled and linked with the > different tools, too, or they'd be incompatible, wouldn't they? > >> I realize some of these are quite old and out of use, but it's pretty simple to produce them all. > > Of course you can contribute whatever installation archives you can create. > I'd rather have a defined set of target platforms and build procedures > for the official release though. So I think we should limit our > set of supported tool sets. What would you suggest that is most likely > to be really useful? > >> > > - I believe there is "new" divergence in m3front between head and >> > Is this relevant for the release? Or can we just ignore it? >> >> I'd have to look, or Tony should say. >> >> To some extent it is tied with if we take NT386 64bit longint. > >> > We still need to add them to the WWW export/display, don't we? >> >> Definitely. > > I'll do that. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > From wagner at elegosoft.com Tue Mar 16 16:28:42 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 16:28:42 +0100 Subject: [M3devel] release status [was something else] In-Reply-To: References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: <20100316162842.qzx2mmd7cc8kockc@mail.elegosoft.com> Quoting Tony Hosking : > Ah, yes, one issue about bringing over m3front changes is that it > also includes the atomics support. I don't think we want to do this > in this release. > So, this argues that we hold off on releasing the NT386 64-bit > LONGINT support for now. > > Thoughts? It's OK by me. You have convinced me that there are too many dependencies and implications. We should be able to start a new RC build in a few days then. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 16 16:33:29 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 15:33:29 +0000 Subject: [M3devel] cvsup Message-ID: Daniel thank you for the bug report. Thank you for suggesting strace. I used strace and compared working vs. non-working. And started added RTIO everywhere. You can also use cvsup -f to slightly simplify -- one fork instead of two. gdb set follow mode didn't seem to help. I almost have it nailed down. in CVSup, FSServer.m3, this code: FINALLY IF isChild THEN SigHandler.ShutDown(); <== ELSE SigHandler.Unblock(); END; END; which runs fairly early, never returns in the child. It ends up here: PROCEDURE ChangeState(d: Dispatcher; state: State) = (* Ask the dispatcher thread to change to a new state, and wait until it has made the change. *) BEGIN LOCK d.mu DO d.desiredState := state; IF d.state # d.desiredState THEN IF d.state = State.Running THEN (* Send dummy signal 0 to wake up the dispatcher. *) Catch(0); ELSE Thread.Signal(d.changeState); END; WHILE d.state # d.desiredState DO Thread.Wait(d.mu, d.stateChanged); <== this never returns END; END; END; END ChangeState; It's a bit wierd to be mixing fork() and Modula-3 Thread? Or maybe it is ok? See, they are asking another process, from the fork() point of view, to change the state. It does so, but the write is private. Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? sbrk is maybe shared? mmap(anon|private) is not. Right now I have #ifndef apple sbrk #else mmap(anon|shared) #endif and it gets further. Hit an assertion failure in pthread. I'll try again. Cleanup all my RTIO. Maybe this notion of using fork() is just not supportable? In either case...you could paint it as an m3core problem, but ?it won't affect much code?. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: cvsup Date: Tue, 16 Mar 2010 12:32:49 +0000 I can reproduce the cvsup problem. An important point to consider is that cvsupd forks a server? Maybe that messes up initialization? I'll dig further. I think these steps are complete. I'll test them on a second clean system. You don't need any real data. jay at xlin2:~$ cat cvsupd.cfg *default tag=. *default host=xlin2. *default prefix=/home/jay *default base=/home/jay/var/db *default release=cvs delete use-rel-suffix compress src-all mkdir -p $HOME/var/db mkdir -p $HOME/data/cvsupd pkill cvsupd # cleanup any previous You don't really need any data. cvsup/cvsupd will issue an error in the "working" case, and hang in the non-working case. run the server: jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd 2010.03.16 05:24:25 PDT [22555]: CVSup server started 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 2010.03.16 05:24:25 PDT [22555]: Ready to service requests run the client: /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg Parsing supfile "/home/jay/cvsupd.cfg" Connecting to xlin2. Connected to xlin2. Server software version: 2010-03-05 Negotiating file attribute support Exchanging collection information Server message: Unknown collection "src-all" Establishing multiplexed-mode data connection Running Skipping collection src-all/cvs Shutting down connection to server Finished successfully it is able to talk to the server, and then fails. Ok -- at least it talked to the server. The server then exits too. I think that is correct (read the usage on the -C option). Then try with -C. The bug says -C 2. Let's use -C 1. They behave the same for our purposes. Just add -C 1 to the above server. Run the same client. The client hangs -- it never connects to the server. I reproduced this on LINUXLIBC6 and PPC_LINUX. Maybe more soon. I'm just rebuilding first. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 16:43:52 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 11:43:52 -0400 Subject: [M3devel] cvsup In-Reply-To: References: Message-ID: <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> mmap is strongly preferred over sbrk. I'd need to understand the control flow here in and across the processes, but yes, generally, mixing pthreads with fork may require significant care. On 16 Mar 2010, at 11:33, Jay K wrote: > Daniel thank you for the bug report. > Thank you for suggesting strace. I used strace > and compared working vs. non-working. > And started added RTIO everywhere. > You can also use cvsup -f to slightly simplify -- one fork instead of two. > gdb set follow mode didn't seem to help. > > > I almost have it nailed down. > > > in CVSup, FSServer.m3, this code: > > FINALLY > IF isChild THEN > SigHandler.ShutDown(); <== > ELSE > SigHandler.Unblock(); > END; > END; > > > which runs fairly early, never returns in the child. > > > It ends up here: > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > (* Ask the dispatcher thread to change to a new state, and wait until > it has made the change. *) > BEGIN > LOCK d.mu DO > d.desiredState := state; > IF d.state # d.desiredState THEN > IF d.state = State.Running THEN > (* Send dummy signal 0 to wake up the dispatcher. *) > Catch(0); > ELSE > Thread.Signal(d.changeState); > END; > WHILE d.state # d.desiredState DO > Thread.Wait(d.mu, d.stateChanged); <== this never returns > END; > END; > END; > END ChangeState; > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > Or maybe it is ok? > > > See, they are asking another process, from the fork() point of view, to change the state. > It does so, but the write is private. > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > sbrk is maybe shared? mmap(anon|private) is not. > > Right now I have > #ifndef apple > sbrk > #else > mmap(anon|shared) > #endif > > > and it gets further. > Hit an assertion failure in pthread. > I'll try again. > Cleanup all my RTIO. > > > Maybe this notion of using fork() is just not supportable? > > > In either case...you could paint it as an m3core problem, but > ?it won't affect much code?. > > > - Jay > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Subject: cvsup > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > I can reproduce the cvsup problem. > An important point to consider is that cvsupd forks a server? > Maybe that messes up initialization? > I'll dig further. > > > I think these steps are complete. > I'll test them on a second clean system. > You don't need any real data. > > > jay at xlin2:~$ cat cvsupd.cfg > *default tag=. > *default host=xlin2. > *default prefix=/home/jay > *default base=/home/jay/var/db > *default release=cvs delete use-rel-suffix compress > > src-all > > > mkdir -p $HOME/var/db > mkdir -p $HOME/data/cvsupd > pkill cvsupd # cleanup any previous > > > You don't really need any data. > cvsup/cvsupd will issue an error in the "working" case, and > hang in the non-working case. > > > > > run the server: > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > run the client: > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > Parsing supfile "/home/jay/cvsupd.cfg" > Connecting to xlin2. > Connected to xlin2. > Server software version: 2010-03-05 > Negotiating file attribute support > Exchanging collection information > Server message: Unknown collection "src-all" > Establishing multiplexed-mode data connection > Running > Skipping collection src-all/cvs > Shutting down connection to server > Finished successfully > > it is able to talk to the server, and then fails. > Ok -- at least it talked to the server. > > The server then exits too. > I think that is correct (read the usage on the -C option). > > > Then try with -C. > The bug says -C 2. Let's use -C 1. > They behave the same for our purposes. > > > Just add -C 1 to the above server. > Run the same client. > The client hangs -- it never connects to the server. > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > Maybe more soon. > I'm just rebuilding first. > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 16 16:51:42 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 15:51:42 +0000 Subject: [M3devel] release status [was something else] In-Reply-To: <20100316162842.qzx2mmd7cc8kockc@mail.elegosoft.com> References: , , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , , , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com>, , , <20100316162842.qzx2mmd7cc8kockc@mail.elegosoft.com> Message-ID: The "experiment" would also probably have to grow ADDRESS. And then for interop I'd have to say, like HANDLE = BITS 32 FOR ADDRESS, if that is allowed. And then..it gets confused..where would I get "32" from? Compiler would have to truncate/extend pointers. Not sure it is willing to. Another good theoretical route is, indeed widen all the pointers, and then #ifdef the C code to take UINT64 and cast to pointers..and convert structs back/forth..but for Win32 we generally don't have such C code as we do for Posix. Darn. I'll think about it more later but maybe it doesn't really work out. - Jay > Date: Tue, 16 Mar 2010 16:28:42 +0100 > From: wagner at elegosoft.com > To: hosking at cs.purdue.edu > CC: jay.krell at cornell.edu; m3devel at elegosoft.com > Subject: Re: [M3devel] release status [was something else] > > Quoting Tony Hosking : > > > Ah, yes, one issue about bringing over m3front changes is that it > > also includes the atomics support. I don't think we want to do this > > in this release. > > So, this argues that we hold off on releasing the NT386 64-bit > > LONGINT support for now. > > > > Thoughts? > > It's OK by me. You have convinced me that there are too many dependencies > and implications. > > We should be able to start a new RC build in a few days then. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 16 17:10:15 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 16:10:15 +0000 Subject: [M3devel] cvsup In-Reply-To: <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> References: , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> Message-ID: I could have sworn I had not seen hanging with sbrk or map_shared, but I can't get that now. I'll dig a bit more..maybe not today. To be clear, this isn't even the usual fork+exec. This is fork + do work + exit. We can probably fix it by having it create a Modula-3 thread. The first fork (daemonize) is probably fine. It is the per-client one that I think is a problem. I'd like to find a theory as to why it ever worked, maybe see it work, and then fix it differently. Maybe sbrk + user threads? (Can anyone claim to have seen cvsup server work with kernel threads?) - Jay From: hosking at cs.purdue.edu Date: Tue, 16 Mar 2010 11:43:52 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup mmap is strongly preferred over sbrk. I'd need to understand the control flow here in and across the processes, but yes, generally, mixing pthreads with fork may require significant care. On 16 Mar 2010, at 11:33, Jay K wrote: Daniel thank you for the bug report. Thank you for suggesting strace. I used strace and compared working vs. non-working. And started added RTIO everywhere. You can also use cvsup -f to slightly simplify -- one fork instead of two. gdb set follow mode didn't seem to help. I almost have it nailed down. in CVSup, FSServer.m3, this code: FINALLY IF isChild THEN SigHandler.ShutDown(); <== ELSE SigHandler.Unblock(); END; END; which runs fairly early, never returns in the child. It ends up here: PROCEDURE ChangeState(d: Dispatcher; state: State) = (* Ask the dispatcher thread to change to a new state, and wait until it has made the change. *) BEGIN LOCK d.mu DO d.desiredState := state; IF d.state # d.desiredState THEN IF d.state = State.Running THEN (* Send dummy signal 0 to wake up the dispatcher. *) Catch(0); ELSE Thread.Signal(d.changeState); END; WHILE d.state # d.desiredState DO Thread.Wait(d.mu, d.stateChanged); <== this never returns END; END; END; END ChangeState; It's a bit wierd to be mixing fork() and Modula-3 Thread? Or maybe it is ok? See, they are asking another process, from the fork() point of view, to change the state. It does so, but the write is private. Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? sbrk is maybe shared? mmap(anon|private) is not. Right now I have #ifndef apple sbrk #else mmap(anon|shared) #endif and it gets further. Hit an assertion failure in pthread. I'll try again. Cleanup all my RTIO. Maybe this notion of using fork() is just not supportable? In either case...you could paint it as an m3core problem, but ?it won't affect much code?. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: cvsup Date: Tue, 16 Mar 2010 12:32:49 +0000 I can reproduce the cvsup problem. An important point to consider is that cvsupd forks a server? Maybe that messes up initialization? I'll dig further. I think these steps are complete. I'll test them on a second clean system. You don't need any real data. jay at xlin2:~$ cat cvsupd.cfg *default tag=. *default host=xlin2. *default prefix=/home/jay *default base=/home/jay/var/db *default release=cvs delete use-rel-suffix compress src-all mkdir -p $HOME/var/db mkdir -p $HOME/data/cvsupd pkill cvsupd # cleanup any previous You don't really need any data. cvsup/cvsupd will issue an error in the "working" case, and hang in the non-working case. run the server: jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd 2010.03.16 05:24:25 PDT [22555]: CVSup server started 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 2010.03.16 05:24:25 PDT [22555]: Ready to service requests run the client: /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg Parsing supfile "/home/jay/cvsupd.cfg" Connecting to xlin2. Connected to xlin2. Server software version: 2010-03-05 Negotiating file attribute support Exchanging collection information Server message: Unknown collection "src-all" Establishing multiplexed-mode data connection Running Skipping collection src-all/cvs Shutting down connection to server Finished successfully it is able to talk to the server, and then fails. Ok -- at least it talked to the server. The server then exits too. I think that is correct (read the usage on the -C option). Then try with -C. The bug says -C 2. Let's use -C 1. They behave the same for our purposes. Just add -C 1 to the above server. Run the same client. The client hangs -- it never connects to the server. I reproduced this on LINUXLIBC6 and PPC_LINUX. Maybe more soon. I'm just rebuilding first. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 16 17:35:40 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 17:35:40 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> Message-ID: <20100316173540.7cgvahkys44ggw4o@mail.elegosoft.com> Quoting Jay K : > I could have sworn I had not seen hanging with sbrk or map_shared, > but I can't get that now. > > I'll dig a bit more..maybe not today. > > To be clear, this isn't even the usual fork+exec. This is fork + do > work + exit. Well, that's how processes on Unix are usually used for scaling. > We can probably fix it by having it create a Modula-3 thread. Hm. That might work, but wouldn't really be the intention. There should be several server processes each with several threads. > The first fork (daemonize) is probably fine. It is the per-client > one that I think is a problem. > > I'd like to find a theory as to why it ever worked, maybe see it > work, and then fix it differently. > > Maybe sbrk + user threads? > > (Can anyone claim to have seen cvsup server work with kernel threads?) No. I think we always used older binaries at Elego. We should be able to make it work though ;-) Olaf > - Jay > > > > From: hosking at cs.purdue.edu > Date: Tue, 16 Mar 2010 11:43:52 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > mmap is strongly preferred over sbrk. > > > I'd need to understand the control flow here in and across the > processes, but yes, generally, mixing pthreads with fork may require > significant care. > > > > On 16 Mar 2010, at 11:33, Jay K wrote: > > Daniel thank you for the bug report. > Thank you for suggesting strace. I used strace > and compared working vs. non-working. > And started added RTIO everywhere. > You can also use cvsup -f to slightly simplify -- one fork instead of two. > gdb set follow mode didn't seem to help. > > > I almost have it nailed down. > > > in CVSup, FSServer.m3, this code: > > FINALLY > IF isChild THEN > SigHandler.ShutDown(); <== > ELSE > SigHandler.Unblock(); > END; > END; > > > which runs fairly early, never returns in the child. > > > It ends up here: > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > (* Ask the dispatcher thread to change to a new state, and wait until > it has made the change. *) > BEGIN > LOCK d.mu DO > d.desiredState := state; > IF d.state # d.desiredState THEN > IF d.state = State.Running THEN > (* Send dummy signal 0 to wake up the dispatcher. *) > Catch(0); > ELSE > Thread.Signal(d.changeState); > END; > WHILE d.state # d.desiredState DO > Thread.Wait(d.mu, d.stateChanged); <== this never returns > END; > END; > END; > END ChangeState; > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > Or maybe it is ok? > > > See, they are asking another process, from the fork() point of view, > to change the state. > It does so, but the write is private. > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > sbrk is maybe shared? mmap(anon|private) is not. > > Right now I have > #ifndef apple > sbrk > #else > mmap(anon|shared) > #endif > > > and it gets further. > Hit an assertion failure in pthread. > I'll try again. > Cleanup all my RTIO. > > > Maybe this notion of using fork() is just not supportable? > > > In either case...you could paint it as an m3core problem, but > ?it won't affect much code?. > > > - Jay > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Subject: cvsup > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > I can reproduce the cvsup problem. > An important point to consider is that cvsupd forks a server? > Maybe that messes up initialization? > I'll dig further. > > > I think these steps are complete. > I'll test them on a second clean system. > You don't need any real data. > > > jay at xlin2:~$ cat cvsupd.cfg > *default tag=. > *default host=xlin2. > *default prefix=/home/jay > *default base=/home/jay/var/db > *default release=cvs delete use-rel-suffix compress > > src-all > > > mkdir -p $HOME/var/db > mkdir -p $HOME/data/cvsupd > pkill cvsupd # cleanup any previous > > > You don't really need any data. > cvsup/cvsupd will issue an error in the "working" case, and > hang in the non-working case. > > > > > run the server: > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > run the client: > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > Parsing supfile "/home/jay/cvsupd.cfg" > Connecting to xlin2. > Connected to xlin2. > Server software version: 2010-03-05 > Negotiating file attribute support > Exchanging collection information > Server message: Unknown collection "src-all" > Establishing multiplexed-mode data connection > Running > Skipping collection src-all/cvs > Shutting down connection to server > Finished successfully > > it is able to talk to the server, and then fails. > Ok -- at least it talked to the server. > > The server then exits too. > I think that is correct (read the usage on the -C option). > > > Then try with -C. > The bug says -C 2. Let's use -C 1. > They behave the same for our purposes. > > > Just add -C 1 to the above server. > Run the same client. > The client hangs -- it never connects to the server. > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > Maybe more soon. > I'm just rebuilding first. > > > - Jay > > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 16 17:38:00 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 16:38:00 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, Message-ID: User threads is apparently sufficient to let it work -- no change to sbrk vs. mmap. You end up with a wierd mix of shared and not shared data, right? I'll try to make it use pthreads and Modula-3 threads and not fork(). - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Tue, 16 Mar 2010 16:10:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup I could have sworn I had not seen hanging with sbrk or map_shared, but I can't get that now. I'll dig a bit more..maybe not today. To be clear, this isn't even the usual fork+exec. This is fork + do work + exit. We can probably fix it by having it create a Modula-3 thread. The first fork (daemonize) is probably fine. It is the per-client one that I think is a problem. I'd like to find a theory as to why it ever worked, maybe see it work, and then fix it differently. Maybe sbrk + user threads? (Can anyone claim to have seen cvsup server work with kernel threads?) - Jay From: hosking at cs.purdue.edu Date: Tue, 16 Mar 2010 11:43:52 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup mmap is strongly preferred over sbrk. I'd need to understand the control flow here in and across the processes, but yes, generally, mixing pthreads with fork may require significant care. On 16 Mar 2010, at 11:33, Jay K wrote: Daniel thank you for the bug report. Thank you for suggesting strace. I used strace and compared working vs. non-working. And started added RTIO everywhere. You can also use cvsup -f to slightly simplify -- one fork instead of two. gdb set follow mode didn't seem to help. I almost have it nailed down. in CVSup, FSServer.m3, this code: FINALLY IF isChild THEN SigHandler.ShutDown(); <== ELSE SigHandler.Unblock(); END; END; which runs fairly early, never returns in the child. It ends up here: PROCEDURE ChangeState(d: Dispatcher; state: State) = (* Ask the dispatcher thread to change to a new state, and wait until it has made the change. *) BEGIN LOCK d.mu DO d.desiredState := state; IF d.state # d.desiredState THEN IF d.state = State.Running THEN (* Send dummy signal 0 to wake up the dispatcher. *) Catch(0); ELSE Thread.Signal(d.changeState); END; WHILE d.state # d.desiredState DO Thread.Wait(d.mu, d.stateChanged); <== this never returns END; END; END; END ChangeState; It's a bit wierd to be mixing fork() and Modula-3 Thread? Or maybe it is ok? See, they are asking another process, from the fork() point of view, to change the state. It does so, but the write is private. Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? sbrk is maybe shared? mmap(anon|private) is not. Right now I have #ifndef apple sbrk #else mmap(anon|shared) #endif and it gets further. Hit an assertion failure in pthread. I'll try again. Cleanup all my RTIO. Maybe this notion of using fork() is just not supportable? In either case...you could paint it as an m3core problem, but ?it won't affect much code?. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: cvsup Date: Tue, 16 Mar 2010 12:32:49 +0000 I can reproduce the cvsup problem. An important point to consider is that cvsupd forks a server? Maybe that messes up initialization? I'll dig further. I think these steps are complete. I'll test them on a second clean system. You don't need any real data. jay at xlin2:~$ cat cvsupd.cfg *default tag=. *default host=xlin2. *default prefix=/home/jay *default base=/home/jay/var/db *default release=cvs delete use-rel-suffix compress src-all mkdir -p $HOME/var/db mkdir -p $HOME/data/cvsupd pkill cvsupd # cleanup any previous You don't really need any data. cvsup/cvsupd will issue an error in the "working" case, and hang in the non-working case. run the server: jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd 2010.03.16 05:24:25 PDT [22555]: CVSup server started 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 2010.03.16 05:24:25 PDT [22555]: Ready to service requests run the client: /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg Parsing supfile "/home/jay/cvsupd.cfg" Connecting to xlin2. Connected to xlin2. Server software version: 2010-03-05 Negotiating file attribute support Exchanging collection information Server message: Unknown collection "src-all" Establishing multiplexed-mode data connection Running Skipping collection src-all/cvs Shutting down connection to server Finished successfully it is able to talk to the server, and then fails. Ok -- at least it talked to the server. The server then exits too. I think that is correct (read the usage on the -C option). Then try with -C. The bug says -C 2. Let's use -C 1. They behave the same for our purposes. Just add -C 1 to the above server. Run the same client. The client hangs -- it never connects to the server. I reproduced this on LINUXLIBC6 and PPC_LINUX. Maybe more soon. I'm just rebuilding first. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 17:42:22 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 12:42:22 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: References: , , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , , , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com>, , , <20100316162842.qzx2mmd7cc8kockc@mail.elegosoft.com> Message-ID: <1C2120B5-5EC7-4850-8704-B37EFA882493@cs.purdue.edu> I think this would be a huge mess. Not worth pursuing. On 16 Mar 2010, at 11:51, Jay K wrote: > The "experiment" would also probably have to grow ADDRESS. > And then for interop I'd have to say, like HANDLE = BITS 32 FOR ADDRESS, if that is allowed. > And then..it gets confused..where would I get "32" from? > Compiler would have to truncate/extend pointers. Not sure it is willing to. > > Another good theoretical route is, indeed widen all the pointers, and then #ifdef the C code to take UINT64 and cast to pointers..and convert structs back/forth..but for Win32 we generally don't have such C code as we do for Posix. Darn. > I'll think about it more later but maybe it doesn't really work out. > > > - Jay > > > > Date: Tue, 16 Mar 2010 16:28:42 +0100 > > From: wagner at elegosoft.com > > To: hosking at cs.purdue.edu > > CC: jay.krell at cornell.edu; m3devel at elegosoft.com > > Subject: Re: [M3devel] release status [was something else] > > > > Quoting Tony Hosking : > > > > > Ah, yes, one issue about bringing over m3front changes is that it > > > also includes the atomics support. I don't think we want to do this > > > in this release. > > > So, this argues that we hold off on releasing the NT386 64-bit > > > LONGINT support for now. > > > > > > Thoughts? > > > > It's OK by me. You have convinced me that there are too many dependencies > > and implications. > > > > We should be able to start a new RC build in a few days then. > > > > Olaf > > -- > > Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 17:43:41 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 12:43:41 -0400 Subject: [M3devel] cvsup In-Reply-To: References: , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> Message-ID: <68E3ECCB-9F5E-4E4C-B8C2-FF1C14B32448@cs.purdue.edu> I thought I did a year or so back. But not certain. On 16 Mar 2010, at 12:10, Jay K wrote: > I could have sworn I had not seen hanging with sbrk or map_shared, but I can't get that now. > I'll dig a bit more..maybe not today. > To be clear, this isn't even the usual fork+exec. This is fork + do work + exit. > We can probably fix it by having it create a Modula-3 thread. > The first fork (daemonize) is probably fine. It is the per-client one that I think is a problem. > I'd like to find a theory as to why it ever worked, maybe see it work, and then fix it differently. > Maybe sbrk + user threads? > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > - Jay > > From: hosking at cs.purdue.edu > Date: Tue, 16 Mar 2010 11:43:52 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > mmap is strongly preferred over sbrk. > > I'd need to understand the control flow here in and across the processes, but yes, generally, mixing pthreads with fork may require significant care. > > On 16 Mar 2010, at 11:33, Jay K wrote: > > Daniel thank you for the bug report. > Thank you for suggesting strace. I used strace > and compared working vs. non-working. > And started added RTIO everywhere. > You can also use cvsup -f to slightly simplify -- one fork instead of two. > gdb set follow mode didn't seem to help. > > > I almost have it nailed down. > > > in CVSup, FSServer.m3, this code: > > FINALLY > IF isChild THEN > SigHandler.ShutDown(); <== > ELSE > SigHandler.Unblock(); > END; > END; > > > which runs fairly early, never returns in the child. > > > It ends up here: > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > (* Ask the dispatcher thread to change to a new state, and wait until > it has made the change. *) > BEGIN > LOCK d.mu DO > d.desiredState := state; > IF d.state # d.desiredState THEN > IF d.state = State.Running THEN > (* Send dummy signal 0 to wake up the dispatcher. *) > Catch(0); > ELSE > Thread.Signal(d.changeState); > END; > WHILE d.state # d.desiredState DO > Thread.Wait(d.mu, d.stateChanged); <== this never returns > END; > END; > END; > END ChangeState; > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > Or maybe it is ok? > > > See, they are asking another process, from the fork() point of view, to change the state. > It does so, but the write is private. > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > sbrk is maybe shared? mmap(anon|private) is not. > > Right now I have > #ifndef apple > sbrk > #else > mmap(anon|shared) > #endif > > > and it gets further. > Hit an assertion failure in pthread. > I'll try again. > Cleanup all my RTIO. > > > Maybe this notion of using fork() is just not supportable? > > > In either case...you could paint it as an m3core problem, but > ?it won't affect much code?. > > > - Jay > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Subject: cvsup > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > I can reproduce the cvsup problem. > An important point to consider is that cvsupd forks a server? > Maybe that messes up initialization? > I'll dig further. > > > I think these steps are complete. > I'll test them on a second clean system. > You don't need any real data. > > > jay at xlin2:~$ cat cvsupd.cfg > *default tag=. > *default host=xlin2. > *default prefix=/home/jay > *default base=/home/jay/var/db > *default release=cvs delete use-rel-suffix compress > > src-all > > > mkdir -p $HOME/var/db > mkdir -p $HOME/data/cvsupd > pkill cvsupd # cleanup any previous > > > You don't really need any data. > cvsup/cvsupd will issue an error in the "working" case, and > hang in the non-working case. > > > > > run the server: > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > run the client: > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > Parsing supfile "/home/jay/cvsupd.cfg" > Connecting to xlin2. > Connected to xlin2. > Server software version: 2010-03-05 > Negotiating file attribute support > Exchanging collection information > Server message: Unknown collection "src-all" > Establishing multiplexed-mode data connection > Running > Skipping collection src-all/cvs > Shutting down connection to server > Finished successfully > > it is able to talk to the server, and then fails. > Ok -- at least it talked to the server. > > The server then exits too. > I think that is correct (read the usage on the -C option). > > > Then try with -C. > The bug says -C 2. Let's use -C 1. > They behave the same for our purposes. > > > Just add -C 1 to the above server. > Run the same client. > The client hangs -- it never connects to the server. > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > Maybe more soon. > I'm just rebuilding first. > > > - Jay > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 16 18:07:38 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 18:07:38 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, Message-ID: <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> Quoting Jay K : > > User threads is apparently sufficient to let it work -- no change to > sbrk vs. mmap. > > You end up with a wierd mix of shared and not shared data, right? > > I'll try to make it use pthreads and Modula-3 threads and not fork(). I cannot really follow you here. Why should fork not work for threaded processes? The kernel should duplicate all threads and all memory regions, right? I don't seem to get the connection you make to sbrk vs. mmap. Can you explain? Using a thread for the process fork will probably not work contrary to my former statement that it might, as each client process is expected to have multiple threads. And the address spaces of the server processes should be separate for security and robustness reasons, as they are each connected to different clients, and one client's server crash must not impact the other sessions. Or am I misunderstanding what you intend to do? Olaf > - Jay > > > > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Tue, 16 Mar 2010 16:10:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > > > I could have sworn I had not seen hanging with sbrk or map_shared, > but I can't get that now. > I'll dig a bit more..maybe not today. > To be clear, this isn't even the usual fork+exec. This is fork + do > work + exit. > We can probably fix it by having it create a Modula-3 thread. > The first fork (daemonize) is probably fine. It is the per-client > one that I think is a problem. > I'd like to find a theory as to why it ever worked, maybe see it > work, and then fix it differently. > Maybe sbrk + user threads? > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > - Jay > > > > From: hosking at cs.purdue.edu > Date: Tue, 16 Mar 2010 11:43:52 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > mmap is strongly preferred over sbrk. > > > I'd need to understand the control flow here in and across the > processes, but yes, generally, mixing pthreads with fork may require > significant care. > > > > On 16 Mar 2010, at 11:33, Jay K wrote: > > Daniel thank you for the bug report. > Thank you for suggesting strace. I used strace > and compared working vs. non-working. > And started added RTIO everywhere. > You can also use cvsup -f to slightly simplify -- one fork instead of two. > gdb set follow mode didn't seem to help. > > > I almost have it nailed down. > > > in CVSup, FSServer.m3, this code: > > FINALLY > IF isChild THEN > SigHandler.ShutDown(); <== > ELSE > SigHandler.Unblock(); > END; > END; > > > which runs fairly early, never returns in the child. > > > It ends up here: > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > (* Ask the dispatcher thread to change to a new state, and wait until > it has made the change. *) > BEGIN > LOCK d.mu DO > d.desiredState := state; > IF d.state # d.desiredState THEN > IF d.state = State.Running THEN > (* Send dummy signal 0 to wake up the dispatcher. *) > Catch(0); > ELSE > Thread.Signal(d.changeState); > END; > WHILE d.state # d.desiredState DO > Thread.Wait(d.mu, d.stateChanged); <== this never returns > END; > END; > END; > END ChangeState; > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > Or maybe it is ok? > > > See, they are asking another process, from the fork() point of view, > to change the state. > It does so, but the write is private. > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > sbrk is maybe shared? mmap(anon|private) is not. > > Right now I have > #ifndef apple > sbrk > #else > mmap(anon|shared) > #endif > > > and it gets further. > Hit an assertion failure in pthread. > I'll try again. > Cleanup all my RTIO. > > > Maybe this notion of using fork() is just not supportable? > > > In either case...you could paint it as an m3core problem, but > ?it won't affect much code?. > > > - Jay > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Subject: cvsup > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > I can reproduce the cvsup problem. > An important point to consider is that cvsupd forks a server? > Maybe that messes up initialization? > I'll dig further. > > > I think these steps are complete. > I'll test them on a second clean system. > You don't need any real data. > > > jay at xlin2:~$ cat cvsupd.cfg > *default tag=. > *default host=xlin2. > *default prefix=/home/jay > *default base=/home/jay/var/db > *default release=cvs delete use-rel-suffix compress > > src-all > > > mkdir -p $HOME/var/db > mkdir -p $HOME/data/cvsupd > pkill cvsupd # cleanup any previous > > > You don't really need any data. > cvsup/cvsupd will issue an error in the "working" case, and > hang in the non-working case. > > > > > run the server: > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > run the client: > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > Parsing supfile "/home/jay/cvsupd.cfg" > Connecting to xlin2. > Connected to xlin2. > Server software version: 2010-03-05 > Negotiating file attribute support > Exchanging collection information > Server message: Unknown collection "src-all" > Establishing multiplexed-mode data connection > Running > Skipping collection src-all/cvs > Shutting down connection to server > Finished successfully > > it is able to talk to the server, and then fails. > Ok -- at least it talked to the server. > > The server then exits too. > I think that is correct (read the usage on the -C option). > > > Then try with -C. > The bug says -C 2. Let's use -C 1. > They behave the same for our purposes. > > > Just add -C 1 to the above server. > Run the same client. > The client hangs -- it never connects to the server. > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > Maybe more soon. > I'm just rebuilding first. > > > - Jay > > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 16 18:43:40 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 17:43:40 +0000 Subject: [M3devel] cvsup In-Reply-To: <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> Message-ID: With fork, which data is copy-on-write vs. shared-write? mmap has a flag, so it is up to each caller. You end up with a mix. Whether or not you get one forked thread or all of them I believe varies. Solaris has "fork1()". There is no obvious answer. It just goes to show how wierd fork() is. I'm not sure any longer there is a connection to mmap/sbrk. It might be just user threads vs. not. Though there again there is a question as to which data is shared-write vs. copy-on-write. The server should not crash. It is written in an optionally save language, after all. More so, that is I believe a much more larger more difficult fix to apply. Using processes for such separation is deemed overkill, depending on whose story you believe. I'll have to think about it more. - Jay > Date: Tue, 16 Mar 2010 18:07:38 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > Quoting Jay K : > > > > > User threads is apparently sufficient to let it work -- no change to > > sbrk vs. mmap. > > > > You end up with a wierd mix of shared and not shared data, right? > > > > I'll try to make it use pthreads and Modula-3 threads and not fork(). > > I cannot really follow you here. Why should fork not work for > threaded processes? The kernel should duplicate all threads and all > memory regions, right? > > I don't seem to get the connection you make to sbrk vs. mmap. > Can you explain? > > Using a thread for the process fork will probably not work contrary > to my former statement that it might, as each client process is > expected to have multiple threads. > > And the address spaces of the server processes should be separate > for security and robustness reasons, as they are each connected to > different clients, and one client's server crash must not impact the > other sessions. > > Or am I misunderstanding what you intend to do? > > Olaf > > > - Jay > > > > > > > > > > From: jay.krell at cornell.edu > > To: hosking at cs.purdue.edu > > Date: Tue, 16 Mar 2010 16:10:15 +0000 > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] cvsup > > > > > > > > I could have sworn I had not seen hanging with sbrk or map_shared, > > but I can't get that now. > > I'll dig a bit more..maybe not today. > > To be clear, this isn't even the usual fork+exec. This is fork + do > > work + exit. > > We can probably fix it by having it create a Modula-3 thread. > > The first fork (daemonize) is probably fine. It is the per-client > > one that I think is a problem. > > I'd like to find a theory as to why it ever worked, maybe see it > > work, and then fix it differently. > > Maybe sbrk + user threads? > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > > > - Jay > > > > > > > > From: hosking at cs.purdue.edu > > Date: Tue, 16 Mar 2010 11:43:52 -0400 > > To: jay.krell at cornell.edu > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] cvsup > > > > mmap is strongly preferred over sbrk. > > > > > > I'd need to understand the control flow here in and across the > > processes, but yes, generally, mixing pthreads with fork may require > > significant care. > > > > > > > > On 16 Mar 2010, at 11:33, Jay K wrote: > > > > Daniel thank you for the bug report. > > Thank you for suggesting strace. I used strace > > and compared working vs. non-working. > > And started added RTIO everywhere. > > You can also use cvsup -f to slightly simplify -- one fork instead of two. > > gdb set follow mode didn't seem to help. > > > > > > I almost have it nailed down. > > > > > > in CVSup, FSServer.m3, this code: > > > > FINALLY > > IF isChild THEN > > SigHandler.ShutDown(); <== > > ELSE > > SigHandler.Unblock(); > > END; > > END; > > > > > > which runs fairly early, never returns in the child. > > > > > > It ends up here: > > > > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > > (* Ask the dispatcher thread to change to a new state, and wait until > > it has made the change. *) > > BEGIN > > LOCK d.mu DO > > d.desiredState := state; > > IF d.state # d.desiredState THEN > > IF d.state = State.Running THEN > > (* Send dummy signal 0 to wake up the dispatcher. *) > > Catch(0); > > ELSE > > Thread.Signal(d.changeState); > > END; > > WHILE d.state # d.desiredState DO > > Thread.Wait(d.mu, d.stateChanged); <== this never returns > > END; > > END; > > END; > > END ChangeState; > > > > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > > Or maybe it is ok? > > > > > > See, they are asking another process, from the fork() point of view, > > to change the state. > > It does so, but the write is private. > > > > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > > sbrk is maybe shared? mmap(anon|private) is not. > > > > Right now I have > > #ifndef apple > > sbrk > > #else > > mmap(anon|shared) > > #endif > > > > > > and it gets further. > > Hit an assertion failure in pthread. > > I'll try again. > > Cleanup all my RTIO. > > > > > > Maybe this notion of using fork() is just not supportable? > > > > > > In either case...you could paint it as an m3core problem, but > > ?it won't affect much code?. > > > > > > - Jay > > > > > > > > From: jay.krell at cornell.edu > > To: m3devel at elegosoft.com > > Subject: cvsup > > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > > > I can reproduce the cvsup problem. > > An important point to consider is that cvsupd forks a server? > > Maybe that messes up initialization? > > I'll dig further. > > > > > > I think these steps are complete. > > I'll test them on a second clean system. > > You don't need any real data. > > > > > > jay at xlin2:~$ cat cvsupd.cfg > > *default tag=. > > *default host=xlin2. > > *default prefix=/home/jay > > *default base=/home/jay/var/db > > *default release=cvs delete use-rel-suffix compress > > > > src-all > > > > > > mkdir -p $HOME/var/db > > mkdir -p $HOME/data/cvsupd > > pkill cvsupd # cleanup any previous > > > > > > You don't really need any data. > > cvsup/cvsupd will issue an error in the "working" case, and > > hang in the non-working case. > > > > > > > > > > run the server: > > > > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > > > > run the client: > > > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > > > > Parsing supfile "/home/jay/cvsupd.cfg" > > Connecting to xlin2. > > Connected to xlin2. > > Server software version: 2010-03-05 > > Negotiating file attribute support > > Exchanging collection information > > Server message: Unknown collection "src-all" > > Establishing multiplexed-mode data connection > > Running > > Skipping collection src-all/cvs > > Shutting down connection to server > > Finished successfully > > > > it is able to talk to the server, and then fails. > > Ok -- at least it talked to the server. > > > > The server then exits too. > > I think that is correct (read the usage on the -C option). > > > > > > Then try with -C. > > The bug says -C 2. Let's use -C 1. > > They behave the same for our purposes. > > > > > > Just add -C 1 to the above server. > > Run the same client. > > The client hangs -- it never connects to the server. > > > > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > > Maybe more soon. > > I'm just rebuilding first. > > > > > > - Jay > > > > > > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 20:32:52 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 15:32:52 -0400 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> Message-ID: fork should still work with pthreads. Why wouldn't it? Unless there is something weird about the semaphore used to start and stop threads? If the child ends up with the same semaphore then that would be a problem. Perhaps we need to ensure it is replicated it in forked children using pthread_atfork? You could try things on a target like OSX where threads are stopped/started natively (i.e., no semaphore being used). Though, on inspection I see that sem_init is called with pshared == false. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 16 Mar 2010, at 13:43, Jay K wrote: > With fork, which data is copy-on-write vs. shared-write? > mmap has a flag, so it is up to each caller. You end up with a mix. > > > Whether or not you get one forked thread or all of them I believe varies. > Solaris has "fork1()". > There is no obvious answer. > It just goes to show how wierd fork() is. > > > I'm not sure any longer there is a connection to mmap/sbrk. > It might be just user threads vs. not. > Though there again there is a question as to which data is > shared-write vs. copy-on-write. > > > The server should not crash. > It is written in an optionally save language, after all. > More so, that is I believe a much more larger more difficult fix to apply. > Using processes for such separation is deemed overkill, depending > on whose story you believe. > I'll have to think about it more. > > > - Jay > > > Date: Tue, 16 Mar 2010 18:07:38 +0100 > > From: wagner at elegosoft.com > > To: m3devel at elegosoft.com > > Subject: Re: [M3devel] cvsup > > > > Quoting Jay K : > > > > > > > > User threads is apparently sufficient to let it work -- no change to > > > sbrk vs. mmap. > > > > > > You end up with a wierd mix of shared and not shared data, right? > > > > > > I'll try to make it use pthreads and Modula-3 threads and not fork(). > > > > I cannot really follow you here. Why should fork not work for > > threaded processes? The kernel should duplicate all threads and all > > memory regions, right? > > > > I don't seem to get the connection you make to sbrk vs. mmap. > > Can you explain? > > > > Using a thread for the process fork will probably not work contrary > > to my former statement that it might, as each client process is > > expected to have multiple threads. > > > > And the address spaces of the server processes should be separate > > for security and robustness reasons, as they are each connected to > > different clients, and one client's server crash must not impact the > > other sessions. > > > > Or am I misunderstanding what you intend to do? > > > > Olaf > > > > > - Jay > > > > > > > > > > > > > > > From: jay.krell at cornell.edu > > > To: hosking at cs.purdue.edu > > > Date: Tue, 16 Mar 2010 16:10:15 +0000 > > > CC: m3devel at elegosoft.com > > > Subject: Re: [M3devel] cvsup > > > > > > > > > > > > I could have sworn I had not seen hanging with sbrk or map_shared, > > > but I can't get that now. > > > I'll dig a bit more..maybe not today. > > > To be clear, this isn't even the usual fork+exec. This is fork + do > > > work + exit. > > > We can probably fix it by having it create a Modula-3 thread. > > > The first fork (daemonize) is probably fine. It is the per-client > > > one that I think is a problem. > > > I'd like to find a theory as to why it ever worked, maybe see it > > > work, and then fix it differently. > > > Maybe sbrk + user threads? > > > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > > > > > - Jay > > > > > > > > > > > > From: hosking at cs.purdue.edu > > > Date: Tue, 16 Mar 2010 11:43:52 -0400 > > > To: jay.krell at cornell.edu > > > CC: m3devel at elegosoft.com > > > Subject: Re: [M3devel] cvsup > > > > > > mmap is strongly preferred over sbrk. > > > > > > > > > I'd need to understand the control flow here in and across the > > > processes, but yes, generally, mixing pthreads with fork may require > > > significant care. > > > > > > > > > > > > On 16 Mar 2010, at 11:33, Jay K wrote: > > > > > > Daniel thank you for the bug report. > > > Thank you for suggesting strace. I used strace > > > and compared working vs. non-working. > > > And started added RTIO everywhere. > > > You can also use cvsup -f to slightly simplify -- one fork instead of two. > > > gdb set follow mode didn't seem to help. > > > > > > > > > I almost have it nailed down. > > > > > > > > > in CVSup, FSServer.m3, this code: > > > > > > FINALLY > > > IF isChild THEN > > > SigHandler.ShutDown(); <== > > > ELSE > > > SigHandler.Unblock(); > > > END; > > > END; > > > > > > > > > which runs fairly early, never returns in the child. > > > > > > > > > It ends up here: > > > > > > > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > > > (* Ask the dispatcher thread to change to a new state, and wait until > > > it has made the change. *) > > > BEGIN > > > LOCK d.mu DO > > > d.desiredState := state; > > > IF d.state # d.desiredState THEN > > > IF d.state = State.Running THEN > > > (* Send dummy signal 0 to wake up the dispatcher. *) > > > Catch(0); > > > ELSE > > > Thread.Signal(d.changeState); > > > END; > > > WHILE d.state # d.desiredState DO > > > Thread.Wait(d.mu, d.stateChanged); <== this never returns > > > END; > > > END; > > > END; > > > END ChangeState; > > > > > > > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > > > Or maybe it is ok? > > > > > > > > > See, they are asking another process, from the fork() point of view, > > > to change the state. > > > It does so, but the write is private. > > > > > > > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > > > sbrk is maybe shared? mmap(anon|private) is not. > > > > > > Right now I have > > > #ifndef apple > > > sbrk > > > #else > > > mmap(anon|shared) > > > #endif > > > > > > > > > and it gets further. > > > Hit an assertion failure in pthread. > > > I'll try again. > > > Cleanup all my RTIO. > > > > > > > > > Maybe this notion of using fork() is just not supportable? > > > > > > > > > In either case...you could paint it as an m3core problem, but > > > ?it won't affect much code?. > > > > > > > > > - Jay > > > > > > > > > > > > From: jay.krell at cornell.edu > > > To: m3devel at elegosoft.com > > > Subject: cvsup > > > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > > > > > I can reproduce the cvsup problem. > > > An important point to consider is that cvsupd forks a server? > > > Maybe that messes up initialization? > > > I'll dig further. > > > > > > > > > I think these steps are complete. > > > I'll test them on a second clean system. > > > You don't need any real data. > > > > > > > > > jay at xlin2:~$ cat cvsupd.cfg > > > *default tag=. > > > *default host=xlin2. > > > *default prefix=/home/jay > > > *default base=/home/jay/var/db > > > *default release=cvs delete use-rel-suffix compress > > > > > > src-all > > > > > > > > > mkdir -p $HOME/var/db > > > mkdir -p $HOME/data/cvsupd > > > pkill cvsupd # cleanup any previous > > > > > > > > > You don't really need any data. > > > cvsup/cvsupd will issue an error in the "working" case, and > > > hang in the non-working case. > > > > > > > > > > > > > > > run the server: > > > > > > > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > > > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > > > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > > > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > > > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > > > > > > > run the client: > > > > > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > > > > > > > Parsing supfile "/home/jay/cvsupd.cfg" > > > Connecting to xlin2. > > > Connected to xlin2. > > > Server software version: 2010-03-05 > > > Negotiating file attribute support > > > Exchanging collection information > > > Server message: Unknown collection "src-all" > > > Establishing multiplexed-mode data connection > > > Running > > > Skipping collection src-all/cvs > > > Shutting down connection to server > > > Finished successfully > > > > > > it is able to talk to the server, and then fails. > > > Ok -- at least it talked to the server. > > > > > > The server then exits too. > > > I think that is correct (read the usage on the -C option). > > > > > > > > > Then try with -C. > > > The bug says -C 2. Let's use -C 1. > > > They behave the same for our purposes. > > > > > > > > > Just add -C 1 to the above server. > > > Run the same client. > > > The client hangs -- it never connects to the server. > > > > > > > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > > > Maybe more soon. > > > I'm just rebuilding first. > > > > > > > > > - Jay > > > > > > > > > > > > > > -- > > Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 16 21:59:12 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 21:59:12 +0100 Subject: [M3devel] make-deb.py Message-ID: <20100316215912.bvvkuei9wk84w8ks@mail.elegosoft.com> I was wondering why there are no Debian or .msi packages, and now have this trace: + type python python is /usr/local/bin/python + [ xAMD64_FREEBSD = xNT386 ] + python /ad0e/home/wagner/work/cm3/scripts/python/make-deb.py /var/tmp/cm3 set CM3_TARGET=AMD64_FREEBSD set CM3_INSTALL=/var/tmp/cm3 set M3CONFIG=/ad0e/home/wagner/work/cm3/m3-sys/cminstall/src/config-no-install/AMD64_FREEBSD Traceback (most recent call last): File "/ad0e/home/wagner/work/cm3/scripts/python/make-deb.py", line 12, in MakeDebianPackage(sys.argv[1], "/usr/local/cm3") TypeError: MakeDebianPackage() takes exactly 4 arguments (2 given) + mv /var/tmp/cm3.deb /var/tmp/cm3-AMD64_FREEBSD-pre-RC5.deb mv: rename /var/tmp/cm3.deb to /var/tmp/cm3-AMD64_FREEBSD-pre-RC5.deb: No such f It seems the scripts are broken or inconsistent in the release branch. Can you please have a look? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From m3 at sol42.com Tue Mar 16 22:19:26 2010 From: m3 at sol42.com (Daniel Solaz) Date: Tue, 16 Mar 2010 22:19:26 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> Message-ID: On 16 Mar 2010, at 20:32, Tony Hosking wrote: > fork should still work with pthreads. Why wouldn't it? Chapter 6.1 of Butenhof's pthreads book deals with this, and well, it is a mess. According to him only the calling thread exists on return from fork() in the child process, but the other pthread *states* (mutexes, conditions, data keys, etc.) are replicated as well. And I'm sure the details vary from system to system. This is the chapter's first sentence: "Avoid using fork in a threaded program (if you can) unless you intend to exec a new program immediately." Regards. -Daniel From rcolebur at SCIRES.COM Wed Mar 17 01:27:39 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Tue, 16 Mar 2010 20:27:39 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: I'm still catching up on my m3devel reading. I was hoping we would put the 64-bit stuff in for NT386. But, I don't want to hold up the release if this is going to be problematic. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Tuesday, March 16, 2010 11:22 AM To: Jay K Cc: m3devel Subject: Re: [M3devel] release status [was something else] Ah, yes, one issue about bringing over m3front changes is that it also includes the atomics support. I don't think we want to do this in this release. So, this argues that we hold off on releasing the NT386 64-bit LONGINT support for now. Thoughts? ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Wed Mar 17 11:35:32 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Wed, 17 Mar 2010 06:35:32 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: References: <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: <20100317103531.GB18479@topoi.pooq.com> On Tue, Mar 16, 2010 at 08:27:39PM -0400, Coleburn, Randy wrote: > I'm still catching up on my m3devel reading. > I was hoping we would put the 64-bit stuff in for NT386. > But, I don't want to hold up the release if this is going to be problematic. > Regards, > Randy Judging from the discussions here, the biggest problem in building a release was that it hadn't been done for some time, and the mechanisms for doing it were all rusty. Well, it shouldn't take as long for hte next one. It's mostly well-oiled by now. -- hendrik From jay.krell at cornell.edu Wed Mar 17 08:47:45 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 07:47:45 +0000 Subject: [M3devel] fork/cvsup Message-ID: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Wed Mar 17 10:19:59 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Wed, 17 Mar 2010 10:19:59 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> Message-ID: <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com> Quoting Daniel Solaz : > On 16 Mar 2010, at 20:32, Tony Hosking wrote: >> fork should still work with pthreads. Why wouldn't it? > Chapter 6.1 of Butenhof's pthreads book deals with this, and well, > it is a mess. According to him only the calling thread exists on > return from fork() in the child process, but the other pthread > *states* (mutexes, conditions, data keys, etc.) are replicated as > well. And I'm sure the details vary from system to system. > This is the chapter's first sentence: "Avoid using fork in a > threaded program (if you can) unless you intend to exec a new > program immediately." Is this about kernel threads or user level threads? Or about a specific implementation of one or the other? I don't know the book, but I always thought about pthreads as being the POSIX specification of threads, and I don't think there's anything in that relating to fork, AFAIR, but that was years ago. Can anyone enlighten us about the POSIX specs and if they define the semantics of fork in presence of threaded processes? Or does it really boil down to the above `avoid it if you can'? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Wed Mar 17 11:07:00 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Wed, 17 Mar 2010 11:07:00 +0100 Subject: [M3devel] Open issues (trac tickets) for cm3 release 5.8 Message-ID: <20100317110700.z1b1sipi80oks8kc@mail.elegosoft.com> There are still a number of open tickets regarding the release in our trac system. Please have a look at https://projects.elego.de/cm3/query?status=resolved&status=reopened&status=assigned&status=analyzed&status=new&status=accepted&group=status&milestone=CM3+release+5.8 I think several of these have already been addressed and can be closed, and others can and should possibly be postponed. Tickets that are assigned to me usually just haven't been assigned to anybody else yet, as I'm the default for that field, so don't ignore them. I can manage these entries, help with information, give you access rights, but haven't the time to do the actual work except in one or two cases. So please try to solve and close as many of the issues as you can before the release. I may also add that analyzing and fixing bugs is always a good starting point for newcomers and those not involved actively so far and just lurking for something to do :-) If you have any questions, don't hesitate to ask me, Olaf PS: Some time ago, there were several requests for trac to coordinate our development. Until now it hasn't really extensively been used; but we should change that. It can really help us to coordinate our work and document changes and their history. It's WiKi can also be used to add various information that is not as formal as tickets and milestones. -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Wed Mar 17 11:42:06 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 10:42:06 +0000 Subject: [M3devel] cvsup In-Reply-To: <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com> References: , , ,,<2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, , , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com> Message-ID: Please start here: http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html "There are at least two serious problems with the semantics of fork() in a multi-threaded program" I've been looking at cvsup a while now. Lots of RTIO.PutText, etc. pthread_atfork usage in RTCollector and ThreadPThread.m3. Reinitialize in the child, mark the gc threads as not being created yet. Been using @M3nogc. It helps. I think I understand why. I have a somewhat wild educated guess as to the problem. It is "fork1" vs. "forkall", sort of. In user threads, when you fork(), you get all the user Modula-3 threads continuing to run in the child. Because their existance is an artifact of a bunch of data that we maintain and gets carried into the new process. With kernel threads (with the exception of using Solaris forkall()), you get just the thread that called fork(). See my reference to the RTCollector/Allocator atfork change, which I show here (not yet commited): PROCEDURE AtForkPrepare() = BEGIN END AtForkPrepare; PROCEDURE AtForkParent() = BEGIN END AtForkParent; PROCEDURE AtForkChild() = VAR r: INTEGER; BEGIN r := Thread.PThreadAtFork(AtForkPrepare, AtForkParent, AtForkChild); <* ASSERT r = 0 *> startedBackground := FALSE; startedForeground := FALSE; END AtForkChild; PROCEDURE Init () = VAR r: INTEGER; BEGIN r := Thread.PThreadAtFork(AtForkPrepare, AtForkParent, AtForkChild); <* ASSERT r = 0 *> Some threads may need to be recreated in the child, some not. The default is not. I think we mainly have to adjust cvsup to recreate its threads. And a little bit of pthread_atfork use in m3core (a bunch of it shown believe, and reinitialization in ThreadPThread.m3). Like the Apple/Darwin warnings, I don't think most libraries can be considered "fork safe". That is you can fork, but only if you exec soon thereafter. Making code "fork safe" I believe equates to reviewing it a bunch and using pthread_atfork selectively. One thing to watch for is if the library creates any worker threads during "initialization" or even "on demand" -- to either recreate them in the child, or note that they don't exist in the child. We can make m3core fork safe, I guess, to satisfy cvsup. Maybe write the code to duplicate all threads in a child, subject to some configuration setting? Something like @M3forkall, but it'd be something that e.g. cvsup would specify when it builds, and then user wouldn't have to list it anywhere. Kind of like the flags for incremental and vm gc that get recorded by the compiler. Anyway, let me see about recreating cvsup's threads manually. Probably by using the Thread.PThreadAtFork I added. I think the initial hang I described fits my theory -- the dispatcher thread doesn't exist in the child, so hang. And then when I avoid that with a local edit I get a hang later. I compared without -C and I see other stuff happening..I think on other threads that fork() is abandoning.. Thread.PThreadAtFork is wierd, but I think reasonable. It will do nothing on user threads and Win32. It will only do anything on PThreads. Libraries can use it to achieve compatibility with pthreads programs that use fork. - Jay > Date: Wed, 17 Mar 2010 10:19:59 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > Quoting Daniel Solaz : > > > On 16 Mar 2010, at 20:32, Tony Hosking wrote: > >> fork should still work with pthreads. Why wouldn't it? > > Chapter 6.1 of Butenhof's pthreads book deals with this, and well, > > it is a mess. According to him only the calling thread exists on > > return from fork() in the child process, but the other pthread > > *states* (mutexes, conditions, data keys, etc.) are replicated as > > well. And I'm sure the details vary from system to system. > > This is the chapter's first sentence: "Avoid using fork in a > > threaded program (if you can) unless you intend to exec a new > > program immediately." > > Is this about kernel threads or user level threads? Or about a specific > implementation of one or the other? > > I don't know the book, but I always thought about pthreads as being the > POSIX specification of threads, and I don't think there's anything > in that relating to fork, AFAIR, but that was years ago. > > Can anyone enlighten us about the POSIX specs and if they define the > semantics of fork in presence of threaded processes? Or does it really > boil down to the above `avoid it if you can'? > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 11:50:31 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 10:50:31 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , ,,,,<2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>,,, , , ,,, , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, , , , , , , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com>, Message-ID: Here's a shorter more direct version. Please start here: http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html "There are at least two serious problems with the semantics of fork() in a multi-threaded program" I have a good theory as to the problem and the fix. Problem: With user threads, when you fork(), all the threads keep running in the new child. Because the data that causes them to exist is carried forward. With kernel threads, they don't, just the caller of fork(). Fix: A few small uses of pthread_atfork both in m3core and cvsup. Abstracted behind Thread.PThreadAtFork that does nothing for user threads and Win32. They would do things like "reinitialize globals" and "recreate worker threads". You can't just call all the module initializers over. That would ultimately I believe reset too much data. ? It might be possible, though, to setjmp, run the module initializers, longjmp. Something like, perhaps, a flag to RTLinker.InitRuntime that causes it to longjmp instead of calling main. However this would still e.g. give the initial thread the wrong stackbase and mess up garbage collection. I'm much more keen on selective use of pthread_atfork, in m3core and cvsup. A more conventional approach is the program to rerun itself with some flags that "push" it fast to "resume" point, but that somewhat defeats the purpose of the fork + do work model -- cheap reuse of already established state. Lingering problem: Any libraries that create worker threads need to use Thread.PThreadAtFork in order to be compatible with the fork + do more work pattern. fork + exec is fine. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 12:02:02 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 11:02:02 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , , , , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , , , , , , , , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, ,,, ,,, ,,, , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com>, , , Message-ID: Another good option might be "RecreateThreadsAfterFork()" that cvsup can call. I'm trying that. If that doesn't work, I'll track down all of cvsup's thread creates. - Jay From: jay.krell at cornell.edu To: wagner at elegosoft.com; m3devel at elegosoft.com Date: Wed, 17 Mar 2010 10:50:31 +0000 Subject: Re: [M3devel] cvsup Here's a shorter more direct version. Please start here: http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html "There are at least two serious problems with the semantics of fork() in a multi-threaded program" I have a good theory as to the problem and the fix. Problem: With user threads, when you fork(), all the threads keep running in the new child. Because the data that causes them to exist is carried forward. With kernel threads, they don't, just the caller of fork(). Fix: A few small uses of pthread_atfork both in m3core and cvsup. Abstracted behind Thread.PThreadAtFork that does nothing for user threads and Win32. They would do things like "reinitialize globals" and "recreate worker threads". You can't just call all the module initializers over. That would ultimately I believe reset too much data. ? It might be possible, though, to setjmp, run the module initializers, longjmp. Something like, perhaps, a flag to RTLinker.InitRuntime that causes it to longjmp instead of calling main. However this would still e.g. give the initial thread the wrong stackbase and mess up garbage collection. I'm much more keen on selective use of pthread_atfork, in m3core and cvsup. A more conventional approach is the program to rerun itself with some flags that "push" it fast to "resume" point, but that somewhat defeats the purpose of the fork + do work model -- cheap reuse of already established state. Lingering problem: Any libraries that create worker threads need to use Thread.PThreadAtFork in order to be compatible with the fork + do more work pattern. fork + exec is fine. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 12:10:43 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 11:10:43 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , , , , , ,,<2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , ,,, , , , , ,,, , ,,, <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, , , , , , , , , , , , , , , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com>, , , , , , Message-ID: For that matter, we can just provide Thread.ForkAll. special implementation on pthreads. Just call Unix.fork() on user threads. No implementation for Win32. - Jay From: jay.krell at cornell.edu To: wagner at elegosoft.com; m3devel at elegosoft.com Date: Wed, 17 Mar 2010 11:02:02 +0000 Subject: Re: [M3devel] cvsup Another good option might be "RecreateThreadsAfterFork()" that cvsup can call. I'm trying that. If that doesn't work, I'll track down all of cvsup's thread creates. - Jay From: jay.krell at cornell.edu To: wagner at elegosoft.com; m3devel at elegosoft.com Date: Wed, 17 Mar 2010 10:50:31 +0000 Subject: Re: [M3devel] cvsup Here's a shorter more direct version. Please start here: http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html "There are at least two serious problems with the semantics of fork() in a multi-threaded program" I have a good theory as to the problem and the fix. Problem: With user threads, when you fork(), all the threads keep running in the new child. Because the data that causes them to exist is carried forward. With kernel threads, they don't, just the caller of fork(). Fix: A few small uses of pthread_atfork both in m3core and cvsup. Abstracted behind Thread.PThreadAtFork that does nothing for user threads and Win32. They would do things like "reinitialize globals" and "recreate worker threads". You can't just call all the module initializers over. That would ultimately I believe reset too much data. ? It might be possible, though, to setjmp, run the module initializers, longjmp. Something like, perhaps, a flag to RTLinker.InitRuntime that causes it to longjmp instead of calling main. However this would still e.g. give the initial thread the wrong stackbase and mess up garbage collection. I'm much more keen on selective use of pthread_atfork, in m3core and cvsup. A more conventional approach is the program to rerun itself with some flags that "push" it fast to "resume" point, but that somewhat defeats the purpose of the fork + do work model -- cheap reuse of already established state. Lingering problem: Any libraries that create worker threads need to use Thread.PThreadAtFork in order to be compatible with the fork + do more work pattern. fork + exec is fine. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Wed Mar 17 12:28:50 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Wed, 17 Mar 2010 12:28:50 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , ,,,,<2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>,,, , , ,,, , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, , , , , , , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com>, Message-ID: <20100317122850.gd98fzkfs0kokgs4@mail.elegosoft.com> Quoting Jay K : > Here's a shorter more direct version. > > Please start here: > http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html > "There are at least two serious problems with the semantics of > fork() in a multi-threaded program" > > I have a good theory as to the problem and the fix. > > Problem: > > With user threads, when you fork(), all the threads keep running in > the new child. > > Because the data that causes them to exist is carried forward. > > With kernel threads, they don't, just the caller of fork(). This is where I seem to have been wrong. I always thought that fork() would indeed duplicate all threads and their execution states. > Fix: > > A few small uses of pthread_atfork both in m3core and cvsup. > > Abstracted behind Thread.PThreadAtFork that does nothing for user > threads and Win32. > > They would do things like "reinitialize globals" and "recreate > worker threads". This sounds reasonable, but is of course not trivial. I think we should at least guarantee that the M3 runtime threads that were running are available again and will do their work. I'm not sure if cvsupd itself will need application extensions, or if the threads handling the streaming protocol are created after the fork. John Polstra used to be on this list and may be able to help. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Wed Mar 17 13:41:23 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 12:41:23 +0000 Subject: [M3devel] cvsup In-Reply-To: <20100317122850.gd98fzkfs0kokgs4@mail.elegosoft.com> References: , , , , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , , , , , , , , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, , , , , , , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com>, , , <20100317122850.gd98fzkfs0kokgs4@mail.elegosoft.com> Message-ID: Olaf, also search the web for fork1, forkall. Solaris has them. You were right for Modula-3 user threads. But not for Win32 or pthreads. I tried something by accident..and it seemed to work. In my attempts to write PThread.ForkAll, which would fork() and then in the child initialize and recreate the threads of the parent (except for the current thread), I commented out the actual creating, leaving in the initialization..it oddly, it worked. I probably had the "ShutDown" hack in cvsup (direct call instead of through the cvsup dispatcher). Perhaps this works because my cvsup test case is too small and doesn't require its other threads. Or maybe it suffices. Still, there are two approaches to a fix. They are both not difficult. One is to provide the overarching function: Thread.ForkAll(). Cvsup would just call that instead of Unix.fork. The other is to provide pthread_atfork wrapper, use it internally a little bit, and have cvsup use it as well to recreate its threads. But I'm not sure it needs to. Modula-3 wouldn't actually have to recreate any threads, just set the booleans in RTCollector to false indicating they haven't been created. Could be that cvsup doesn't require any/many threads to flow from server to forked server, maybe just the one that it tears down right away. - Jay > Date: Wed, 17 Mar 2010 12:28:50 +0100 > From: wagner at elegosoft.com > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] cvsup > > Quoting Jay K : > > > Here's a shorter more direct version. > > > > Please start here: > > http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html > > "There are at least two serious problems with the semantics of > > fork() in a multi-threaded program" > > > > I have a good theory as to the problem and the fix. > > > > Problem: > > > > With user threads, when you fork(), all the threads keep running in > > the new child. > > > > Because the data that causes them to exist is carried forward. > > > > With kernel threads, they don't, just the caller of fork(). > > This is where I seem to have been wrong. I always thought that fork() > would indeed duplicate all threads and their execution states. > > > Fix: > > > > A few small uses of pthread_atfork both in m3core and cvsup. > > > > Abstracted behind Thread.PThreadAtFork that does nothing for user > > threads and Win32. > > > > They would do things like "reinitialize globals" and "recreate > > worker threads". > > This sounds reasonable, but is of course not trivial. > I think we should at least guarantee that the M3 runtime threads > that were running are available again and will do their work. > > I'm not sure if cvsupd itself will need application extensions, or if > the threads handling the streaming protocol are created after the fork. > > John Polstra used to be on this list and may be able to help. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 17 14:28:14 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 17 Mar 2010 09:28:14 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: References: Message-ID: Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such time as one of the exec functions is > called. All APIs, including global data symbols, in any framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 15:01:15 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 14:01:15 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: References: , Message-ID: Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 16:39:59 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 15:39:59 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , Message-ID: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 17 19:01:31 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 17 Mar 2010 14:01:31 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , Message-ID: <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu> Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: > I have something working on Solaris now. > More details after testing on Linux and Darwin. > > - Jay > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Exec what? > You'd have to change the code to carefully reach the same place. > > - Jay > > Subject: Re: [M3devel] fork/cvsup > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 09:28:14 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not just exec in the child? > > On 17 Mar 2010, at 03:47, Jay K wrote: > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such time as one of the exec functions is > called. All APIs, including global data symbols, in any framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 19:13:45 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 18:13:45 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu> References: , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu> Message-ID: --- bad news: It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. Restart manually. Works again. Around 9 times. Then hangs again. That is on Linux/x86 and Solaris/sparc. Doesn't work at all on Mac/amd64, just hangs. --- sketch: m3core uses pthread_atfork to selectively reinitialize Mainly to only have one thread. common Thread.PThreadAtFork is provided for others to do the same It is deliberately in a portable interface. Thread.ReforkThreadAfterProcessFork Is provided for users to restart threads from their child AtFork hander. This is used by the allocator/collector. Thread.ForkProcessAndAllThreads() Is used by "lazy" clients who want to restart all their threads but didn't keep track of them. The runtime can do it for them. This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads or not, depending on if they need their threads restarted. The runtime takes care of its threads either way. --- What'd I'd written up: attached works typically 9 times on Linux and Solaris before server hangs again. No improvement on Darwin, just hangs. Can't see much in debuggers for some reason. There is extra allowance in the m3core change such that users of fork + do work (as opposed to fork + exec) may or may not call ForkAll, depending on if they feel a need for their own threads to be recreated, and if they've kept track of how to recreate them, or just rely on the runtime to know all the threads. There are three runtime threads that are sometimes created in the parent, and if so, recreated in the child. background collector, foreground collector, weak ref thread I'll try to poke at it some more. I'm not sure what is the best way to suspend all threads. I tried a few differnt ways. SuspendOthers LockHeap pthread_mutex_lock various combinations It is deliberate that pthread specific code is in common/Thread.i3. That way code can be portable, at least among the two Posix thread implementations. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:31 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: m3core_atfork.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cvsup_forkall.txt URL: From hosking at cs.purdue.edu Wed Mar 17 19:30:47 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 17 Mar 2010 14:30:47 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu> Message-ID: <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu> I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 14:13, Jay K wrote: > --- > bad news: > It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. > Restart manually. Works again. Around 9 times. Then hangs again. > That is on Linux/x86 and Solaris/sparc. > Doesn't work at all on Mac/amd64, just hangs. > > --- > sketch: > m3core uses pthread_atfork to selectively reinitialize > Mainly to only have one thread. > > > common Thread.PThreadAtFork is provided for others to do the same > It is deliberately in a portable interface. > > > Thread.ReforkThreadAfterProcessFork > Is provided for users to restart threads from their child AtFork hander. > This is used by the allocator/collector. > > > Thread.ForkProcessAndAllThreads() > Is used by "lazy" clients who want to restart all their threads > but didn't keep track of them. The runtime can do it for them. > > > This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads > or not, depending on if they need their threads restarted. > The runtime takes care of its threads either way. > > > --- > What'd I'd written up: > > attached works typically 9 times on Linux and Solaris > before server hangs again. > > > No improvement on Darwin, just hangs. > Can't see much in debuggers for some reason. > > > There is extra allowance in the m3core change such > that users of fork + do work (as opposed to fork + exec) > may or may not call ForkAll, depending on if they > feel a need for their own threads to be recreated, > and if they've kept track of how to recreate them, > or just rely on the runtime to know all the threads. > > > There are three runtime threads that are sometimes > created in the parent, and if so, recreated in the child. > background collector, foreground collector, weak ref thread > > > I'll try to poke at it some more. > > > I'm not sure what is the best way to suspend all threads. > I tried a few differnt ways. > SuspendOthers > LockHeap > pthread_mutex_lock > various combinations > > > It is deliberate that pthread specific code is in common/Thread.i3. > That way code can be portable, at least among the two Posix thread implementations. > > > - Jay > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:31 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Can you sketch the approach you've taken? > > > On 17 Mar 2010, at 11:39, Jay K wrote: > > I have something working on Solaris now. > More details after testing on Linux and Darwin. > > - Jay > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Exec what? > You'd have to change the code to carefully reach the same place. > > - Jay > > Subject: Re: [M3devel] fork/cvsup > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 09:28:14 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not just exec in the child? > > On 17 Mar 2010, at 03:47, Jay K wrote: > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such time as one of the exec functions is > called. All APIs, including global data symbols, in any framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 23:05:25 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 22:05:25 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu> References: , , ,,, , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu> Message-ID: Tony, I don't know. Here is some "argument', but I'm not sure. Adding threads does something different. Such threads would share mutation to global state. I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:30:47 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 14:13, Jay K wrote: --- bad news: It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. Restart manually. Works again. Around 9 times. Then hangs again. That is on Linux/x86 and Solaris/sparc. Doesn't work at all on Mac/amd64, just hangs. --- sketch: m3core uses pthread_atfork to selectively reinitialize Mainly to only have one thread. common Thread.PThreadAtFork is provided for others to do the same It is deliberately in a portable interface. Thread.ReforkThreadAfterProcessFork Is provided for users to restart threads from their child AtFork hander. This is used by the allocator/collector. Thread.ForkProcessAndAllThreads() Is used by "lazy" clients who want to restart all their threads but didn't keep track of them. The runtime can do it for them. This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads or not, depending on if they need their threads restarted. The runtime takes care of its threads either way. --- What'd I'd written up: attached works typically 9 times on Linux and Solaris before server hangs again. No improvement on Darwin, just hangs. Can't see much in debuggers for some reason. There is extra allowance in the m3core change such that users of fork + do work (as opposed to fork + exec) may or may not call ForkAll, depending on if they feel a need for their own threads to be recreated, and if they've kept track of how to recreate them, or just rely on the runtime to know all the threads. There are three runtime threads that are sometimes created in the parent, and if so, recreated in the child. background collector, foreground collector, weak ref thread I'll try to poke at it some more. I'm not sure what is the best way to suspend all threads. I tried a few differnt ways. SuspendOthers LockHeap pthread_mutex_lock various combinations It is deliberate that pthread specific code is in common/Thread.i3. That way code can be portable, at least among the two Posix thread implementations. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:31 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 23:18:37 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 22:18:37 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: References: , ,,,,,,, , , ,,, , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, Message-ID: Maybe an easier approach is to offer a way in quake to select user threads, that then "sticks" automatically through runtime? That either implies build_standalone, or provides a differently named m3core.so? Replacing m3core.so doesn't work, and wouldn't solve this problem. Presumably one would want cvsup to coexist with non-user threads programs. You have expressed not liking these ideas in the past though. It wouldn't necessarily cost even a function pointer indirection for pthreads. That is, it wouldn't necessarily be an @M3userthreads runtime choice. The choice would be made at compile time and the appropriate functions called directly. Well, granted, the easiest way to implement that is "directly" through other functions. Programs that need "fork all" semantics could use user threads. We could alter the types slightly I think so they have the same fingerprints. So that everything wouldn't have to be recompiled, so that it'd be a build time replacement of one .lib or the other. Ie. first fix it so that swapping .so files works. And then make user_threads imply build_standalone. Does that make sense? To repeat: first adjust the two Thread.T to match, if that is possible. Not their code, just the types. And then build m3core twice. And have user threads imply build_standalone. It'd probably just be a matter of adding a few unused fields. I think I'll first try the "limited atfork" idea. That code is in hand already at least to start. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 22:05:25 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Tony, I don't know. Here is some "argument', but I'm not sure. Adding threads does something different. Such threads would share mutation to global state. I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:30:47 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 14:13, Jay K wrote: --- bad news: It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. Restart manually. Works again. Around 9 times. Then hangs again. That is on Linux/x86 and Solaris/sparc. Doesn't work at all on Mac/amd64, just hangs. --- sketch: m3core uses pthread_atfork to selectively reinitialize Mainly to only have one thread. common Thread.PThreadAtFork is provided for others to do the same It is deliberately in a portable interface. Thread.ReforkThreadAfterProcessFork Is provided for users to restart threads from their child AtFork hander. This is used by the allocator/collector. Thread.ForkProcessAndAllThreads() Is used by "lazy" clients who want to restart all their threads but didn't keep track of them. The runtime can do it for them. This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads or not, depending on if they need their threads restarted. The runtime takes care of its threads either way. --- What'd I'd written up: attached works typically 9 times on Linux and Solaris before server hangs again. No improvement on Darwin, just hangs. Can't see much in debuggers for some reason. There is extra allowance in the m3core change such that users of fork + do work (as opposed to fork + exec) may or may not call ForkAll, depending on if they feel a need for their own threads to be recreated, and if they've kept track of how to recreate them, or just rely on the runtime to know all the threads. There are three runtime threads that are sometimes created in the parent, and if so, recreated in the child. background collector, foreground collector, weak ref thread I'll try to poke at it some more. I'm not sure what is the best way to suspend all threads. I tried a few differnt ways. SuspendOthers LockHeap pthread_mutex_lock various combinations It is deliberate that pthread specific code is in common/Thread.i3. That way code can be portable, at least among the two Posix thread implementations. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:31 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 17 23:32:44 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 17 Mar 2010 18:32:44 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , , , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu> Message-ID: <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> What are the expectations in the cvsup child regarding the threads it inherits? On 17 Mar 2010, at 18:05, Jay K wrote: > Tony, I don't know. > Here is some "argument', but I'm not sure. > > > Adding threads does something different. Such threads would share mutation to global state. > I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. > > I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave > people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. > > > There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:30:47 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 17 Mar 2010, at 14:13, Jay K wrote: > > --- > bad news: > It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. > Restart manually. Works again. Around 9 times. Then hangs again. > That is on Linux/x86 and Solaris/sparc. > Doesn't work at all on Mac/amd64, just hangs. > > --- > sketch: > m3core uses pthread_atfork to selectively reinitialize > Mainly to only have one thread. > > > common Thread.PThreadAtFork is provided for others to do the same > It is deliberately in a portable interface. > > > Thread.ReforkThreadAfterProcessFork > Is provided for users to restart threads from their child AtFork hander. > This is used by the allocator/collector. > > > Thread.ForkProcessAndAllThreads() > Is used by "lazy" clients who want to restart all their threads > but didn't keep track of them. The runtime can do it for them. > > > This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads > or not, depending on if they need their threads restarted. > The runtime takes care of its threads either way. > > > --- > What'd I'd written up: > > attached works typically 9 times on Linux and Solaris > before server hangs again. > > > No improvement on Darwin, just hangs. > Can't see much in debuggers for some reason. > > > There is extra allowance in the m3core change such > that users of fork + do work (as opposed to fork + exec) > may or may not call ForkAll, depending on if they > feel a need for their own threads to be recreated, > and if they've kept track of how to recreate them, > or just rely on the runtime to know all the threads. > > > There are three runtime threads that are sometimes > created in the parent, and if so, recreated in the child. > background collector, foreground collector, weak ref thread > > > I'll try to poke at it some more. > > > I'm not sure what is the best way to suspend all threads. > I tried a few differnt ways. > SuspendOthers > LockHeap > pthread_mutex_lock > various combinations > > > It is deliberate that pthread specific code is in common/Thread.i3. > That way code can be portable, at least among the two Posix thread implementations. > > > - Jay > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:31 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Can you sketch the approach you've taken? > > > On 17 Mar 2010, at 11:39, Jay K wrote: > > I have something working on Solaris now. > More details after testing on Linux and Darwin. > > - Jay > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Exec what? > You'd have to change the code to carefully reach the same place. > > - Jay > > Subject: Re: [M3devel] fork/cvsup > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 09:28:14 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not just exec in the child? > > On 17 Mar 2010, at 03:47, Jay K wrote: > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such time as one of the exec functions is > called. All APIs, including global data symbols, in any framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 17 23:40:17 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 17 Mar 2010 18:40:17 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , , , , , , , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, Message-ID: <5EBE34A1-B22E-4E03-AB59-5A7E4ABFAAA9@cs.purdue.edu> That makes little sense. Surely the cvsup server would want to run with proper multi-threading? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 18:18, Jay K wrote: > Maybe an easier approach is to offer a way in quake to select user threads, > that then "sticks" automatically through runtime? > That either implies build_standalone, or provides a differently named m3core.so? > Replacing m3core.so doesn't work, and wouldn't solve this problem. Presumably > one would want cvsup to coexist with non-user threads programs. > > > You have expressed not liking these ideas in the past though. > > > It wouldn't necessarily cost even a function pointer indirection for pthreads. > That is, it wouldn't necessarily be an @M3userthreads runtime choice. > > > The choice would be made at compile time and the appropriate functions called directly. > Well, granted, the easiest way to implement that is "directly" through other functions. > > > Programs that need "fork all" semantics could use user threads. > > > We could alter the types slightly I think so they have the same fingerprints. > So that everything wouldn't have to be recompiled, so that it'd be a build time > replacement of one .lib or the other. Ie. first fix it so that swapping .so files > works. And then make user_threads imply build_standalone. > Does that make sense? > To repeat: first adjust the two Thread.T to match, if that is possible. > Not their code, just the types. > And then build m3core twice. > And have user threads imply build_standalone. > > > It'd probably just be a matter of adding a few unused fields. > > > I think I'll first try the "limited atfork" idea. > That code is in hand already at least to start. > > > - Jay > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 22:05:25 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Tony, I don't know. > Here is some "argument', but I'm not sure. > > > Adding threads does something different. Such threads would share mutation to global state. > I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. > > I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave > people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. > > > There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:30:47 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 17 Mar 2010, at 14:13, Jay K wrote: > > --- > bad news: > It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. > Restart manually. Works again. Around 9 times. Then hangs again. > That is on Linux/x86 and Solaris/sparc. > Doesn't work at all on Mac/amd64, just hangs. > > --- > sketch: > m3core uses pthread_atfork to selectively reinitialize > Mainly to only have one thread. > > > common Thread.PThreadAtFork is provided for others to do the same > It is deliberately in a portable interface. > > > Thread.ReforkThreadAfterProcessFork > Is provided for users to restart threads from their child AtFork hander. > This is used by the allocator/collector. > > > Thread.ForkProcessAndAllThreads() > Is used by "lazy" clients who want to restart all their threads > but didn't keep track of them. The runtime can do it for them. > > > This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads > or not, depending on if they need their threads restarted. > The runtime takes care of its threads either way. > > > --- > What'd I'd written up: > > attached works typically 9 times on Linux and Solaris > before server hangs again. > > > No improvement on Darwin, just hangs. > Can't see much in debuggers for some reason. > > > There is extra allowance in the m3core change such > that users of fork + do work (as opposed to fork + exec) > may or may not call ForkAll, depending on if they > feel a need for their own threads to be recreated, > and if they've kept track of how to recreate them, > or just rely on the runtime to know all the threads. > > > There are three runtime threads that are sometimes > created in the parent, and if so, recreated in the child. > background collector, foreground collector, weak ref thread > > > I'll try to poke at it some more. > > > I'm not sure what is the best way to suspend all threads. > I tried a few differnt ways. > SuspendOthers > LockHeap > pthread_mutex_lock > various combinations > > > It is deliberate that pthread specific code is in common/Thread.i3. > That way code can be portable, at least among the two Posix thread implementations. > > > - Jay > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:31 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Can you sketch the approach you've taken? > > > On 17 Mar 2010, at 11:39, Jay K wrote: > > I have something working on Solaris now. > More details after testing on Linux and Darwin. > > - Jay > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Exec what? > You'd have to change the code to carefully reach the same place. > > - Jay > > Subject: Re: [M3devel] fork/cvsup > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 09:28:14 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not just exec in the child? > > On 17 Mar 2010, at 03:47, Jay K wrote: > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such time as one of the exec functions is > called. All APIs, including global data symbols, in any framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 18 00:04:23 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 23:04:23 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: <5EBE34A1-B22E-4E03-AB59-5A7E4ABFAAA9@cs.purdue.edu> References: , , , , ,,, , , ,,, , ,,, , ,,<553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, ,,, , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, , , , <5EBE34A1-B22E-4E03-AB59-5A7E4ABFAAA9@cs.purdue.edu> Message-ID: [somewhat self follow up] These ideas are not mutually exclusive. It should be easier to switch to user threads. One might imagine "IMPORT UserThread AS Thread" though with that construct, you'd have a system using a mix of each thread type, depending on what a module imported, on a module by module basis. Anyway, let me try something that doesn't involve user threads. Something like what I sent but smaller. I also need to see if user threads exhibit the "only it works 9 times" problem, and if I can debug that either way. It really seems to be a fixed number of times. Hm. I have been prone to say -C 99. I should make sure that isn't interpreted as -C 9. I do sometimes say -C 2. And that's supposed to be sequential clients and I've seen what it does when it hits the limit it prints a specific message (some forms do hit the limit, because the hangs are in child servers and the master server knows they are still running). Hm. I'll have to dig in more/again. I'm still not comfortable with my use of SuspendOthers. Is it guaranteed where such threads are? I know they are sitting in the signal header, on some platforms, but no guarantee what got interrupted by it, right? They don't have any of the Thread.m3 locks? But they might have some other locks? I guess like the atfork documentation says, for this stuff to work, everyone with a lock has to use atfork. Good point about the server, though it never historically has. We're stuck between user threads used to work for it and kernel threads probably never have. Or it could be my theories are all wrong. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 18:40:17 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup That makes little sense. Surely the cvsup server would want to run with proper multi-threading? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 18:18, Jay K wrote: Maybe an easier approach is to offer a way in quake to select user threads, that then "sticks" automatically through runtime? That either implies build_standalone, or provides a differently named m3core.so? Replacing m3core.so doesn't work, and wouldn't solve this problem. Presumably one would want cvsup to coexist with non-user threads programs. You have expressed not liking these ideas in the past though. It wouldn't necessarily cost even a function pointer indirection for pthreads. That is, it wouldn't necessarily be an @M3userthreads runtime choice. The choice would be made at compile time and the appropriate functions called directly. Well, granted, the easiest way to implement that is "directly" through other functions. Programs that need "fork all" semantics could use user threads. We could alter the types slightly I think so they have the same fingerprints. So that everything wouldn't have to be recompiled, so that it'd be a build time replacement of one .lib or the other. Ie. first fix it so that swapping .so files works. And then make user_threads imply build_standalone. Does that make sense? To repeat: first adjust the two Thread.T to match, if that is possible. Not their code, just the types. And then build m3core twice. And have user threads imply build_standalone. It'd probably just be a matter of adding a few unused fields. I think I'll first try the "limited atfork" idea. That code is in hand already at least to start. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 22:05:25 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Tony, I don't know. Here is some "argument', but I'm not sure. Adding threads does something different. Such threads would share mutation to global state. I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:30:47 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 14:13, Jay K wrote: --- bad news: It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. Restart manually. Works again. Around 9 times. Then hangs again. That is on Linux/x86 and Solaris/sparc. Doesn't work at all on Mac/amd64, just hangs. --- sketch: m3core uses pthread_atfork to selectively reinitialize Mainly to only have one thread. common Thread.PThreadAtFork is provided for others to do the same It is deliberately in a portable interface. Thread.ReforkThreadAfterProcessFork Is provided for users to restart threads from their child AtFork hander. This is used by the allocator/collector. Thread.ForkProcessAndAllThreads() Is used by "lazy" clients who want to restart all their threads but didn't keep track of them. The runtime can do it for them. This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads or not, depending on if they need their threads restarted. The runtime takes care of its threads either way. --- What'd I'd written up: attached works typically 9 times on Linux and Solaris before server hangs again. No improvement on Darwin, just hangs. Can't see much in debuggers for some reason. There is extra allowance in the m3core change such that users of fork + do work (as opposed to fork + exec) may or may not call ForkAll, depending on if they feel a need for their own threads to be recreated, and if they've kept track of how to recreate them, or just rely on the runtime to know all the threads. There are three runtime threads that are sometimes created in the parent, and if so, recreated in the child. background collector, foreground collector, weak ref thread I'll try to poke at it some more. I'm not sure what is the best way to suspend all threads. I tried a few differnt ways. SuspendOthers LockHeap pthread_mutex_lock various combinations It is deliberate that pthread specific code is in common/Thread.i3. That way code can be portable, at least among the two Posix thread implementations. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:31 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 18 00:12:09 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 23:12:09 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> References: , , , ,,, , ,,, ,,, , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, , <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> Message-ID: I don't know. I know there is one thread that it merely shuts right down in the child. It does that by queueing a message to it. The initial problem you see in the current code is it hangs trying to do that. You can basically just remove that code (except then it won't work with user threads probably!). The problem you hit after that is ThreadPThread.m3 getting, I forget the errno, but pthread_kill complains that you give it a nonexistant thread. That's presumably because the child process has inherited the parent's data as to existant threads. "limited atfork" addresses that. "limited atfork" means, a subset of the diff I sent. So the next things for me to try: - verify user threads doesn't fail after 9 also - verify that 9 isn't associated with "-C 99". - assuming no to both of those, try "limited atfork" and remove the code to shutdown the (nonexistant) dispatcher thread. If that works, almost done. Only remaining part would be to expose a boolean from Thread.i3 so cvsup could make the right choice. There might be a way to structure the cvsup code to work either way and not have to know. Something like signaling the thread ahead of time that it might be going away, and unsignaling only in the parent. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 18:32:44 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup What are the expectations in the cvsup child regarding the threads it inherits? On 17 Mar 2010, at 18:05, Jay K wrote: Tony, I don't know. Here is some "argument', but I'm not sure. Adding threads does something different. Such threads would share mutation to global state. I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:30:47 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 14:13, Jay K wrote: --- bad news: It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. Restart manually. Works again. Around 9 times. Then hangs again. That is on Linux/x86 and Solaris/sparc. Doesn't work at all on Mac/amd64, just hangs. --- sketch: m3core uses pthread_atfork to selectively reinitialize Mainly to only have one thread. common Thread.PThreadAtFork is provided for others to do the same It is deliberately in a portable interface. Thread.ReforkThreadAfterProcessFork Is provided for users to restart threads from their child AtFork hander. This is used by the allocator/collector. Thread.ForkProcessAndAllThreads() Is used by "lazy" clients who want to restart all their threads but didn't keep track of them. The runtime can do it for them. This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads or not, depending on if they need their threads restarted. The runtime takes care of its threads either way. --- What'd I'd written up: attached works typically 9 times on Linux and Solaris before server hangs again. No improvement on Darwin, just hangs. Can't see much in debuggers for some reason. There is extra allowance in the m3core change such that users of fork + do work (as opposed to fork + exec) may or may not call ForkAll, depending on if they feel a need for their own threads to be recreated, and if they've kept track of how to recreate them, or just rely on the runtime to know all the threads. There are three runtime threads that are sometimes created in the parent, and if so, recreated in the child. background collector, foreground collector, weak ref thread I'll try to poke at it some more. I'm not sure what is the best way to suspend all threads. I tried a few differnt ways. SuspendOthers LockHeap pthread_mutex_lock various combinations It is deliberate that pthread specific code is in common/Thread.i3. That way code can be portable, at least among the two Posix thread implementations. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:31 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Wed Mar 17 13:29:43 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Wed, 17 Mar 2010 13:29:43 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> Message-ID: <1268828983.2906.0.camel@faramir.m3w.org> Yes, I can. I am building it regularly, from version to version, at least few times a year. And it also worked with my ancient kernel thread implementation. Was one of stress tests. On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > (Can anyone claim to have seen cvsup server work with kernel threads?) -- Dragi?a Duri? From jay.krell at cornell.edu Thu Mar 18 11:09:38 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 18 Mar 2010 10:09:38 +0000 Subject: [M3devel] cvsup In-Reply-To: <1268828983.2906.0.camel@faramir.m3w.org> References: ,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , <1268828983.2906.0.camel@faramir.m3w.org> Message-ID: Dragisha, Really? The server? With the -C flag and/or -f? Evidence is very very very very very good as to what is going on here. It is all related to "fork1" vs. "forkall". fork1 is the overwhelming usual behavior (Solaris has an option), and basically *any* library that uses pthreads is obligated to use pthread_atfork, be it a C library or the Modula-3 library, etc.. The application cannot do things, unless the library exposes its internal locks. The Posix spec for pthread_atfork explains the problem. Consider C code that links in Modula-3 code. C code is fairly free to use the fork + do work model. It isn't very common, but it does exist, and people have gone to extra length to keep it viable. bash and ccache I believe both use this model. Granted, if you had your own kernel thread implementation, it might have addressed this. I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions. The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork. I want to still try out the "bigger" change, but with the simpler lock/unlock. And test on Solaris and Darwin. I think for SuspendOthers to not work is not surprising. The threads can still hold a few locks even if suspended, I'm pretty sure. The point is to fork in a "controlled" fashion -- all locks held, and in the right order, so that both parent and child can release them. Taking "all" the locks in Prepare is more reliable. I do wonder if really "all" locks need to be taken. I only take the static ones declared in PThreadThread.c. - Jay > Subject: Re: [M3devel] cvsup > From: dragisha at m3w.org > To: jay.krell at cornell.edu > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > Yes, I can. I am building it regularly, from version to version, at > least few times a year. And it also worked with my ancient kernel thread > implementation. Was one of stress tests. > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Mar 18 13:56:11 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 18 Mar 2010 13:56:11 +0100 Subject: [M3devel] cvsup In-Reply-To: References: ,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> , ,<1268828983.2906.0.camel@faramir.m3w.org> Message-ID: <1268916971.2586.3.camel@faramir.m3w.org> Client, it's what I am using. And it's dead as we speak. Last worked before I pulled/built RC4 On Thu, 2010-03-18 at 10:09 +0000, Jay K wrote: > Dragisha, Really? The server? With the -C flag and/or -f? > > Evidence is very very very very very good as to what is going on here. > It is all related to "fork1" vs. "forkall". > fork1 is the overwhelming usual behavior (Solaris has an option), > and basically *any* library that uses pthreads is obligated to use > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > The application cannot do things, unless > the library exposes its internal locks. The Posix spec for > pthread_atfork explains the problem. > > Consider C code that links in Modula-3 code. > C code is fairly free to use the fork + do work model. It isn't very > common, > but it does exist, and people have gone to extra length to keep it > viable. > bash and ccache I believe both use this model. > > > Granted, if you had your own kernel thread implementation, it might > have addressed this. > > > I have a version now that has served over 1000 requests, similar to > what I sent, but a bit reduced. The main change was I just called > pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead > of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just > Linux. This version also doesn't expose the ForkProcessAndAllThreads > functions. > The client has to restart its threads, or in the cvsupd case, don't > depend on the dispatcher thread to survive fork. > I want to still try out the "bigger" change, but with the simpler > lock/unlock. > And test on Solaris and Darwin. > > > I think for SuspendOthers to not work is not surprising. > The threads can still hold a few locks even if suspended, I'm pretty > sure. > The point is to fork in a "controlled" fashion -- all locks held, and > in > the right order, so that both parent and child can release them. > > > Taking "all" the locks in Prepare is more reliable. > > > I do wonder if really "all" locks need to be taken. > I only take the static ones declared in PThreadThread.c. > > - Jay > > > > Subject: Re: [M3devel] cvsup > > From: dragisha at m3w.org > > To: jay.krell at cornell.edu > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > Yes, I can. I am building it regularly, from version to version, at > > least few times a year. And it also worked with my ancient kernel > thread > > implementation. Was one of stress tests. > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > (Can anyone claim to have seen cvsup server work with kernel > threads?) > > -- > > Dragi?a Duri? > > -- Dragi?a Duri? From wagner at elegosoft.com Thu Mar 18 15:25:10 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Thu, 18 Mar 2010 15:25:10 +0100 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , ,,, , ,,, ,,, , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, , <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> Message-ID: <20100318152510.nr94m5wi1wkw0048@mail.elegosoft.com> I'm not able to follow your discussion completely now, but some information about CVSup can be found at http://www.cvsup.org/ http://www.cvsup.org/howsofast.html AFAIK it uses the traditional Unix daemon pattern to fork a new server process for each client connection, which then creates threads for the streaming protocol. It should be possible to change the pattern to use threads only, but I'd rather like it if we could support the traditional Unix daemon pattern in a general form, too. Modula-3 is a systems programming language after all. Olaf Quoting Jay K : > > I don't know. I know there is one thread that it merely shuts right > down in the child. > > It does that by queueing a message to it. > > The initial problem you see in the current code is it hangs trying > to do that. > > > > > > You can basically just remove that code (except then it won't work > > with user threads probably!). > > > > > > The problem you hit after that is ThreadPThread.m3 getting, > > I forget the errno, but pthread_kill complains that you give > > it a nonexistant thread. That's presumably because the child > > process has inherited the parent's data as to existant threads. > > "limited atfork" addresses that. "limited atfork" means, a subset > > of the diff I sent. > > > > > > So the next things for me to try: > > - verify user threads doesn't fail after 9 also > > - verify that 9 isn't associated with "-C 99". > > - assuming no to both of those, try "limited atfork" and > > remove the code to shutdown the (nonexistant) dispatcher > > thread. If that works, almost done. Only remaining part would > > be to expose a boolean from Thread.i3 so cvsup could > > make the right choice. There might be a way to structure > > the cvsup code to work either way and not have to know. > > Something like signaling the thread ahead of time that it might > > be going away, and unsignaling only in the parent. > > > > > > - Jay > > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 18:32:44 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > What are the expectations in the cvsup child regarding the threads > it inherits? > > > > > On 17 Mar 2010, at 18:05, Jay K wrote: > > Tony, I don't know. > Here is some "argument', but I'm not sure. > > > Adding threads does something different. Such threads would share > mutation to global state. > I'm not a big fan of this model, but fork lets you establish some > perhaps expensive to establish state, then share it cheaply among a > bunch of future threads/processes, that may make their own local > modifications to it. One would have to read the cvsup code a bunch > to determine what it actually does and requires. > > I do suspect there is a general solution. Leaving anyone who uses > platform specific functions to fend for themselves seems a bit > unfair. Which functions to we abtract away in m3ore vs. which do we > leave > people to use on their own? And does that list change much in time? > Well, infinity isn't possible either, granted. And we've only seen > one program so far that cares, we shouldn't spend too much just for > one program. > > > There may be a smaller related fix, where m3core internally uses > atfork, but doesn't expose ForkAll to the client. I know cvsup has > the dispatcher thread that it expects to be inherited by children, > however all it does with it is queue a request to it to shut itself > down. In that way, ForkAll is a waste -- it recreates a thread, only > so the client can shut it down. I can pursue that more. > > > - Jay > > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:30:47 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > I don't think there is a "general" solution to this that should be > applied to the thread library. Modula-3 does not mandate any > support for fork! It is not part of the language. If a program > relies on platform-specific interfaces then it must be the one to > handle situations arising from the problem. Why does cvsup need to > fork in the first place? Surely it can simply add threads to handle > clients as they arrive? > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > On 17 Mar 2010, at 14:13, Jay K wrote: > > --- > bad news: > It doesn't completely work. It works a bunch of times in a row, like > 9, then hangs. > Restart manually. Works again. Around 9 times. Then hangs again. > That is on Linux/x86 and Solaris/sparc. > Doesn't work at all on Mac/amd64, just hangs. > > --- > sketch: > m3core uses pthread_atfork to selectively reinitialize > Mainly to only have one thread. > > > common Thread.PThreadAtFork is provided for others to do the same > It is deliberately in a portable interface. > > > Thread.ReforkThreadAfterProcessFork > Is provided for users to restart threads from their child AtFork hander. > This is used by the allocator/collector. > > > Thread.ForkProcessAndAllThreads() > Is used by "lazy" clients who want to restart all their threads > but didn't keep track of them. The runtime can do it for them. > > > This allows for "fork + do work" folks do call or not call > ForkProcessAndAllThreads > or not, depending on if they need their threads restarted. > The runtime takes care of its threads either way. > > > --- > What'd I'd written up: > > attached works typically 9 times on Linux and Solaris > before server hangs again. > > > No improvement on Darwin, just hangs. > Can't see much in debuggers for some reason. > > > There is extra allowance in the m3core change such > that users of fork + do work (as opposed to fork + exec) > may or may not call ForkAll, depending on if they > feel a need for their own threads to be recreated, > and if they've kept track of how to recreate them, > or just rely on the runtime to know all the threads. > > > There are three runtime threads that are sometimes > created in the parent, and if so, recreated in the child. > background collector, foreground collector, weak ref thread > > > I'll try to poke at it some more. > > > I'm not sure what is the best way to suspend all threads. > I tried a few differnt ways. > SuspendOthers > LockHeap > pthread_mutex_lock > various combinations > > > It is deliberate that pthread specific code is in common/Thread.i3. > That way code can be portable, at least among the two Posix thread > implementations. > > > - Jay > > > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:31 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Can you sketch the approach you've taken? > > > > > On 17 Mar 2010, at 11:39, Jay K wrote: > > I have something working on Solaris now. > More details after testing on Linux and Darwin. > > - Jay > > > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Exec what? > You'd have to change the code to carefully reach the same place. > > - Jay > > > > Subject: Re: [M3devel] fork/cvsup > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 09:28:14 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > > > > Why not just exec in the child? > > > On 17 Mar 2010, at 03:47, Jay K wrote: > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be > totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such > time as one of the exec functions is > called. All APIs, including global data symbols, in any > framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe > or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this > situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute > async-signal-safe operations until such time as one of theexec > functions is called. [THR] Fork handlers may be established by > means of the pthread_atfork() function in order to maintain > application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another > thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the > pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a > record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. > mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be > initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon > creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay > > > > > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From hosking at cs.purdue.edu Thu Mar 18 15:31:15 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 18 Mar 2010 10:31:15 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: <20100318152510.nr94m5wi1wkw0048@mail.elegosoft.com> References: , , , , , , , , , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, , <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> <20100318152510.nr94m5wi1wkw0048@mail.elegosoft.com> Message-ID: <318D6C6F-80BE-41F5-80CC-03387905439B@cs.purdue.edu> Agreed. I do seem to recall cvsupd working with system threads way back when I was testing things. On 18 Mar 2010, at 10:25, Olaf Wagner wrote: > I'm not able to follow your discussion completely now, but some > information about CVSup can be found at > > http://www.cvsup.org/ > http://www.cvsup.org/howsofast.html > > AFAIK it uses the traditional Unix daemon pattern to fork a new > server process for each client connection, which then creates threads > for the streaming protocol. > > It should be possible to change the pattern to use threads only, > but I'd rather like it if we could support the traditional Unix > daemon pattern in a general form, too. Modula-3 is a systems > programming language after all. > > Olaf > > Quoting Jay K : > >> >> I don't know. I know there is one thread that it merely shuts right down in the child. >> >> It does that by queueing a message to it. >> >> The initial problem you see in the current code is it hangs trying to do that. >> >> >> >> >> >> You can basically just remove that code (except then it won't work >> >> with user threads probably!). >> >> >> >> >> >> The problem you hit after that is ThreadPThread.m3 getting, >> >> I forget the errno, but pthread_kill complains that you give >> >> it a nonexistant thread. That's presumably because the child >> >> process has inherited the parent's data as to existant threads. >> >> "limited atfork" addresses that. "limited atfork" means, a subset >> >> of the diff I sent. >> >> >> >> >> >> So the next things for me to try: >> >> - verify user threads doesn't fail after 9 also >> >> - verify that 9 isn't associated with "-C 99". >> >> - assuming no to both of those, try "limited atfork" and >> >> remove the code to shutdown the (nonexistant) dispatcher >> >> thread. If that works, almost done. Only remaining part would >> >> be to expose a boolean from Thread.i3 so cvsup could >> >> make the right choice. There might be a way to structure >> >> the cvsup code to work either way and not have to know. >> >> Something like signaling the thread ahead of time that it might >> >> be going away, and unsignaling only in the parent. >> >> >> >> >> >> - Jay >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 18:32:44 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> What are the expectations in the cvsup child regarding the threads it inherits? >> >> >> >> >> On 17 Mar 2010, at 18:05, Jay K wrote: >> >> Tony, I don't know. >> Here is some "argument', but I'm not sure. >> >> >> Adding threads does something different. Such threads would share mutation to global state. >> I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. >> >> I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave >> people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. >> >> >> There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. >> >> >> - Jay >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:30:47 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? >> >> >> >> >> Antony Hosking | Associate Professor | Computer Science | Purdue University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> On 17 Mar 2010, at 14:13, Jay K wrote: >> >> --- >> bad news: >> It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. >> Restart manually. Works again. Around 9 times. Then hangs again. >> That is on Linux/x86 and Solaris/sparc. >> Doesn't work at all on Mac/amd64, just hangs. >> >> --- >> sketch: >> m3core uses pthread_atfork to selectively reinitialize >> Mainly to only have one thread. >> >> >> common Thread.PThreadAtFork is provided for others to do the same >> It is deliberately in a portable interface. >> >> >> Thread.ReforkThreadAfterProcessFork >> Is provided for users to restart threads from their child AtFork hander. >> This is used by the allocator/collector. >> >> >> Thread.ForkProcessAndAllThreads() >> Is used by "lazy" clients who want to restart all their threads >> but didn't keep track of them. The runtime can do it for them. >> >> >> This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads >> or not, depending on if they need their threads restarted. >> The runtime takes care of its threads either way. >> >> >> --- >> What'd I'd written up: >> >> attached works typically 9 times on Linux and Solaris >> before server hangs again. >> >> >> No improvement on Darwin, just hangs. >> Can't see much in debuggers for some reason. >> >> >> There is extra allowance in the m3core change such >> that users of fork + do work (as opposed to fork + exec) >> may or may not call ForkAll, depending on if they >> feel a need for their own threads to be recreated, >> and if they've kept track of how to recreate them, >> or just rely on the runtime to know all the threads. >> >> >> There are three runtime threads that are sometimes >> created in the parent, and if so, recreated in the child. >> background collector, foreground collector, weak ref thread >> >> >> I'll try to poke at it some more. >> >> >> I'm not sure what is the best way to suspend all threads. >> I tried a few differnt ways. >> SuspendOthers >> LockHeap >> pthread_mutex_lock >> various combinations >> >> >> It is deliberate that pthread specific code is in common/Thread.i3. >> That way code can be portable, at least among the two Posix thread implementations. >> >> >> - Jay >> >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:01:31 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> Can you sketch the approach you've taken? >> >> >> >> >> On 17 Mar 2010, at 11:39, Jay K wrote: >> >> I have something working on Solaris now. >> More details after testing on Linux and Darwin. >> >> - Jay >> >> >> >> From: jay.krell at cornell.edu >> To: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:01:15 +0000 >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> Exec what? >> You'd have to change the code to carefully reach the same place. >> >> - Jay >> >> >> >> Subject: Re: [M3devel] fork/cvsup >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 09:28:14 -0400 >> CC: m3devel at elegosoft.com >> To: jay.krell at cornell.edu >> >> >> >> >> Why not just exec in the child? >> >> >> On 17 Mar 2010, at 03:47, Jay K wrote: >> >> http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html >> >> >> There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself >> self to only executing async-signal safe operations until such time as one of the exec functions is >> called. All APIs, including global data symbols, in any framework or library should be assumed to be >> unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use >> these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. >> self. >> >> >> http://www.opengroup.org/onlinepubs/000095399/functions/fork.html >> >> Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. >> >> >> I've run through a few theories so far. >> Current thinking is related to what Tony said: >> use pthread_atfork: >> in prepare, stopworld >> in parent, resumeworld >> You don't want the child to be mid-gc for example, on another thread. Or mid-anything. >> in child, reinitialize -- current thread is the only thread >> >> >> Also in the cvsup code, ShutDown should just call DoShutDown immediately. >> I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. >> pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. >> >> >> I don't have a theory as to why user threads work. >> >> >> I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). >> I was expecting more cases to act like mmap(shared), but none did, only it. >> >> >> I experimented with having mutexes and condition variables be initialize up front instead of on-demand. >> Via changing cvsup to lock/unlock or broadcast immediately upon creating them. >> On the theory that might let them work across process. >> That didn't make a difference. >> >> >> - Jay >> >> >> >> >> > > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > From jay.krell at cornell.edu Thu Mar 18 16:45:11 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 18 Mar 2010 15:45:11 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <1268828983.2906.0.camel@faramir.m3w.org>, Message-ID: To repeat myself: > and basically *any* library that uses pthreads is obligated to use > pthread_atfork, be it a C library or the Modula-3 library, etc.. This is the crux of the matter. We are being "bad pthread citizens" by not using pthread_atfork. Granted, we are probably in large company. There is a smaller fix and a larger fix, but it seems very clear we should take one of them. The small fix is: common/Thread.i3: add PThreadAtFork It does nothing on posix/nt. It calls pthread_atfork on pthread. In RTCollector.m3 give a child handler that stores FALSE in the three booleans that indicate the three threads have started. No prepare or parent handler I believe. Though I should check for global locks there. Probably the locks in ThreadPThread suffice? In ThreadPThread.m3 provide three handlers: prepare: lock "everything", at least the 5 static locks. I might try walking all the threads and locking them too. parent: unlock everything child: unlock everything and reinitialize the globals Note that the atfork handlers run in many more programs than cvsup. They would be used also with fork + exec. Perhaps a way to disable that -- no way in Posix, we could just provide a boolean to ourselves. The larger fix would be to provide a common/Thread.i3 function to call fork() *and* recreate all Modula-3 threads in the child process. That is what I sent out. In the second case, you also then need to allow that the caller may or may not use that function, so you still need atfork handlers, but they have to be slightly different, as the diffs I sent show. I believe strongly this is the way to go. It is how one is a "good pthread citizen". Now, granted, I don't think most libraries are. Witness the caveat about fork() in the Darwin manpage and I think even the Posix spec. But pthread_atfork is meant to solve this. I don't suggest a full audit and add atfork handlers everywhere. But m3core we can at least do, and that should satisfy cvsup. Should check libm3 too. Plus, as long as each module takes reponsibility for itself, it shouldn't be so hard. That is, I don't think the case is all that strong for providing the "forkall" function that is in the diffs I sent. I'll do more testing and send out diffs "later". It could be as long as a week, I have a bunch of things to do. Or maybe just a day. :) I tested a form of this fix + cvsup + user threads and it appears that cvsup can do one small thing and thereby work either way, without knowledge as to the way. That is, it can shutdown the dispatcher "directly", instead of queuing to it. I'm not even sure the dispatcher thread is really needed. As I said, I don't believe the application can handle this all itself. Any library that uses pthreads is obligated to play into the general mechanism. Generally the internals are not exposed for the application to do it. Treating Modula-3 as a closed system in which nobody uses anything outside of or "below" m3core/libm3 I don't think is practical. "applications", arguably defines as "processes", are often composed of fairly disparate bodies of code that don't necessarily expose much to each other. For example, they might each create their own worker threads and not expose them. This is what pthread_atfork is for. Actually Tony, you gave me the lead here. :) There is a problem that m3posixthreads (user) and pthreads semantics diverage, and..I haven't been able to think of an way to fix that. Well, it is actually easy to make "forkall" the default and only behavior actually, on user and pthreads (but not NT). But I don't see a way to make "fork1" the behavior for user threads. You can associate a pid with each thread, and notice when the pid changes, but I don't know which one thread you would keep, and you wouldn't necessarily be able to register pthread_atfork handlers, unless maybe user threads were used only on platforms that actually have pthreads.. And still there's no good way to it on NT I expect. - Jay From: jay.krell at cornell.edu To: dragisha at m3w.org Date: Thu, 18 Mar 2010 10:09:38 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup Dragisha, Really? The server? With the -C flag and/or -f? Evidence is very very very very very good as to what is going on here. It is all related to "fork1" vs. "forkall". fork1 is the overwhelming usual behavior (Solaris has an option), and basically *any* library that uses pthreads is obligated to use pthread_atfork, be it a C library or the Modula-3 library, etc.. The application cannot do things, unless the library exposes its internal locks. The Posix spec for pthread_atfork explains the problem. Consider C code that links in Modula-3 code. C code is fairly free to use the fork + do work model. It isn't very common, but it does exist, and people have gone to extra length to keep it viable. bash and ccache I believe both use this model. Granted, if you had your own kernel thread implementation, it might have addressed this. I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions. The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork. I want to still try out the "bigger" change, but with the simpler lock/unlock. And test on Solaris and Darwin. I think for SuspendOthers to not work is not surprising. The threads can still hold a few locks even if suspended, I'm pretty sure. The point is to fork in a "controlled" fashion -- all locks held, and in the right order, so that both parent and child can release them. Taking "all" the locks in Prepare is more reliable. I do wonder if really "all" locks need to be taken. I only take the static ones declared in PThreadThread.c. - Jay > Subject: Re: [M3devel] cvsup > From: dragisha at m3w.org > To: jay.krell at cornell.edu > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > Yes, I can. I am building it regularly, from version to version, at > least few times a year. And it also worked with my ancient kernel thread > implementation. Was one of stress tests. > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdp at polstra.com Thu Mar 18 17:56:57 2010 From: jdp at polstra.com (John Polstra) Date: Thu, 18 Mar 2010 09:56:57 -0700 Subject: [M3devel] fork/cvsup In-Reply-To: <20100318152510.nr94m5wi1wkw0048@mail.elegosoft.com> References: , , , , , , , , , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, , <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> <20100318152510.nr94m5wi1wkw0048@mail.elegosoft.com> Message-ID: <79503C57-BC05-44E5-B0E2-494639A1FBD7@polstra.com> I don't want to get too involved in this, because it's been years and years since I even glanced at CVSup. (I retired from the software business a couple years ago.) But yes, it forks a new process per client and uses threads for the streaming protocol within each client process. I tried doing it all with threads when I first wrote it, but that turned out to be a bad idea for reasons of robustness. With the all-threads approach, any internal error (assert failure, bounds check, etc.) for any client will kill *all* clients. That's unacceptable from the user's point of view. You are on the right track with this bug. There is some kind of bad interaction between the forks and the threads system. I remember some issues along the same lines when I was developing the software, and I had to find what worked by experiment. At that time there were no standard facilities for using fork within threaded programs. Now that you have switched to using OS-provided threads (a good change, I think), it's not surprising that some problems have cropped up. John On Mar 18, 2010, at 7:25 AM, Olaf Wagner wrote: > I'm not able to follow your discussion completely now, but some > information about CVSup can be found at > > http://www.cvsup.org/ > http://www.cvsup.org/howsofast.html > > AFAIK it uses the traditional Unix daemon pattern to fork a new > server process for each client connection, which then creates threads > for the streaming protocol. > > It should be possible to change the pattern to use threads only, > but I'd rather like it if we could support the traditional Unix > daemon pattern in a general form, too. Modula-3 is a systems > programming language after all. > > Olaf > > Quoting Jay K : > >> >> I don't know. I know there is one thread that it merely shuts >> right down in the child. >> >> It does that by queueing a message to it. >> >> The initial problem you see in the current code is it hangs trying >> to do that. >> >> >> >> >> >> You can basically just remove that code (except then it won't work >> >> with user threads probably!). >> >> >> >> >> >> The problem you hit after that is ThreadPThread.m3 getting, >> >> I forget the errno, but pthread_kill complains that you give >> >> it a nonexistant thread. That's presumably because the child >> >> process has inherited the parent's data as to existant threads. >> >> "limited atfork" addresses that. "limited atfork" means, a subset >> >> of the diff I sent. >> >> >> >> >> >> So the next things for me to try: >> >> - verify user threads doesn't fail after 9 also >> >> - verify that 9 isn't associated with "-C 99". >> >> - assuming no to both of those, try "limited atfork" and >> >> remove the code to shutdown the (nonexistant) dispatcher >> >> thread. If that works, almost done. Only remaining part would >> >> be to expose a boolean from Thread.i3 so cvsup could >> >> make the right choice. There might be a way to structure >> >> the cvsup code to work either way and not have to know. >> >> Something like signaling the thread ahead of time that it might >> >> be going away, and unsignaling only in the parent. >> >> >> >> >> >> - Jay >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 18:32:44 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> What are the expectations in the cvsup child regarding the threads >> it inherits? >> >> >> >> >> On 17 Mar 2010, at 18:05, Jay K wrote: >> >> Tony, I don't know. >> Here is some "argument', but I'm not sure. >> >> >> Adding threads does something different. Such threads would share >> mutation to global state. >> I'm not a big fan of this model, but fork lets you establish some >> perhaps expensive to establish state, then share it cheaply among >> a bunch of future threads/processes, that may make their own >> local modifications to it. One would have to read the cvsup code a >> bunch to determine what it actually does and requires. >> >> I do suspect there is a general solution. Leaving anyone who uses >> platform specific functions to fend for themselves seems a bit >> unfair. Which functions to we abtract away in m3ore vs. which do >> we leave >> people to use on their own? And does that list change much in >> time? Well, infinity isn't possible either, granted. And we've >> only seen one program so far that cares, we shouldn't spend too >> much just for one program. >> >> >> There may be a smaller related fix, where m3core internally uses >> atfork, but doesn't expose ForkAll to the client. I know cvsup has >> the dispatcher thread that it expects to be inherited by children, >> however all it does with it is queue a request to it to shut >> itself down. In that way, ForkAll is a waste -- it recreates a >> thread, only so the client can shut it down. I can pursue that more. >> >> >> - Jay >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:30:47 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> I don't think there is a "general" solution to this that should be >> applied to the thread library. Modula-3 does not mandate any >> support for fork! It is not part of the language. If a program >> relies on platform-specific interfaces then it must be the one to >> handle situations arising from the problem. Why does cvsup need >> to fork in the first place? Surely it can simply add threads to >> handle clients as they arrive? >> >> >> >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> On 17 Mar 2010, at 14:13, Jay K wrote: >> >> --- >> bad news: >> It doesn't completely work. It works a bunch of times in a row, >> like 9, then hangs. >> Restart manually. Works again. Around 9 times. Then hangs again. >> That is on Linux/x86 and Solaris/sparc. >> Doesn't work at all on Mac/amd64, just hangs. >> >> --- >> sketch: >> m3core uses pthread_atfork to selectively reinitialize >> Mainly to only have one thread. >> >> >> common Thread.PThreadAtFork is provided for others to do the same >> It is deliberately in a portable interface. >> >> >> Thread.ReforkThreadAfterProcessFork >> Is provided for users to restart threads from their child AtFork >> hander. >> This is used by the allocator/collector. >> >> >> Thread.ForkProcessAndAllThreads() >> Is used by "lazy" clients who want to restart all their threads >> but didn't keep track of them. The runtime can do it for them. >> >> >> This allows for "fork + do work" folks do call or not call >> ForkProcessAndAllThreads >> or not, depending on if they need their threads restarted. >> The runtime takes care of its threads either way. >> >> >> --- >> What'd I'd written up: >> >> attached works typically 9 times on Linux and Solaris >> before server hangs again. >> >> >> No improvement on Darwin, just hangs. >> Can't see much in debuggers for some reason. >> >> >> There is extra allowance in the m3core change such >> that users of fork + do work (as opposed to fork + exec) >> may or may not call ForkAll, depending on if they >> feel a need for their own threads to be recreated, >> and if they've kept track of how to recreate them, >> or just rely on the runtime to know all the threads. >> >> >> There are three runtime threads that are sometimes >> created in the parent, and if so, recreated in the child. >> background collector, foreground collector, weak ref thread >> >> >> I'll try to poke at it some more. >> >> >> I'm not sure what is the best way to suspend all threads. >> I tried a few differnt ways. >> SuspendOthers >> LockHeap >> pthread_mutex_lock >> various combinations >> >> >> It is deliberate that pthread specific code is in common/Thread.i3. >> That way code can be portable, at least among the two Posix thread >> implementations. >> >> >> - Jay >> >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:01:31 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> Can you sketch the approach you've taken? >> >> >> >> >> On 17 Mar 2010, at 11:39, Jay K wrote: >> >> I have something working on Solaris now. >> More details after testing on Linux and Darwin. >> >> - Jay >> >> >> >> From: jay.krell at cornell.edu >> To: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:01:15 +0000 >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> Exec what? >> You'd have to change the code to carefully reach the same place. >> >> - Jay >> >> >> >> Subject: Re: [M3devel] fork/cvsup >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 09:28:14 -0400 >> CC: m3devel at elegosoft.com >> To: jay.krell at cornell.edu >> >> >> >> >> Why not just exec in the child? >> >> >> On 17 Mar 2010, at 03:47, Jay K wrote: >> >> http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html >> >> >> There are limits to what you can do in the child process. To be >> totally safe you should restrict your-self yourself >> self to only executing async-signal safe operations until such >> time as one of the exec functions is >> called. All APIs, including global data symbols, in any >> framework or library should be assumed to be >> unsafe after a fork() unless explicitly documented to be safe >> or async-signal safe. If you need to use >> these frameworks in the child process, you must exec. In this >> situation it is reasonable to exec your-self. yourself. >> self. >> >> >> http://www.opengroup.org/onlinepubs/000095399/functions/fork.html >> >> Consequently, to avoid errors, the child process may only execute >> async-signal-safe operations until such time as one of theexec >> functions is called. [THR] Fork handlers may be established by >> means of the pthread_atfork() function in order to maintain >> application invariants across fork() calls. >> >> >> I've run through a few theories so far. >> Current thinking is related to what Tony said: >> use pthread_atfork: >> in prepare, stopworld >> in parent, resumeworld >> You don't want the child to be mid-gc for example, on another >> thread. Or mid-anything. >> in child, reinitialize -- current thread is the only thread >> >> >> Also in the cvsup code, ShutDown should just call DoShutDown >> immediately. >> I did that, without m3core changes, and it hits an error in the >> pthread code, signaling a nonexistant thread. >> pthread_atfork/child should address that -- child shouldn't retain >> a record of all the threads in the parent. >> >> >> I don't have a theory as to why user threads work. >> >> >> I experimented with malloc vs. static alloc vs. sbrk vs. >> mmap(private) vs. mmap(shared). >> I was expecting more cases to act like mmap(shared), but none did, >> only it. >> >> >> I experimented with having mutexes and condition variables be >> initialize up front instead of on-demand. >> Via changing cvsup to lock/unlock or broadcast immediately upon >> creating them. >> On the theory that might let them work across process. >> That didn't make a difference. >> >> >> - Jay >> >> >> >> >> > > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, > Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 > 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: > Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: > DE163214194 > From hosking at cs.purdue.edu Thu Mar 18 18:16:03 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 18 Mar 2010 13:16:03 -0400 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <1268828983.2906.0.camel@faramir.m3w.org>, Message-ID: On 18 Mar 2010, at 11:45, Jay K wrote: > To repeat myself: > > > and basically *any* library that uses pthreads is obligated to use > > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > > This is the crux of the matter. > We are being "bad pthread citizens" by not using pthread_atfork. > Granted, we are probably in large company. > > > There is a smaller fix and a larger fix, but it seems very clear we > should take one of them. > > > The small fix is: > common/Thread.i3: add PThreadAtFork Name should probably be something more like Thread.AtFork. > It does nothing on posix/nt. > It calls pthread_atfork on pthread. > > > In RTCollector.m3 give a child handler that stores FALSE in the > three booleans that indicate the three threads have started. > No prepare or parent handler I believe. > Though I should check for global locks there. > Probably the locks in ThreadPThread suffice? I wonder if we should only fork when the collector is turned off? > In ThreadPThread.m3 provide three handlers: > prepare: lock "everything", at least the 5 static locks. > I might try walking all the threads and locking them too. > parent: unlock everything > child: unlock everything and reinitialize the globals > > > Note that the atfork handlers run in many more programs than cvsup. > They would be used also with fork + exec. fork + exec is probably relatively safe already. > Perhaps a way to disable that -- no way in Posix, we could just > provide a boolean to ourselves. Not sure I follow. > The larger fix would be to provide a common/Thread.i3 function > to call fork() *and* recreate all Modula-3 threads in the child process. > That is what I sent out. That's tough to do portably! > In the second case, you also then need to allow that the caller > may or may not use that function, so you still need atfork handlers, > but they have to be slightly different, as the diffs I sent show. > > > I believe strongly this is the way to go. > It is how one is a "good pthread citizen". > Now, granted, I don't think most libraries are. > Witness the caveat about fork() in the Darwin manpage and > I think even the Posix spec. > But pthread_atfork is meant to solve this. > > > I don't suggest a full audit and add atfork handlers everywhere. > But m3core we can at least do, and that should satisfy cvsup. > Should check libm3 too. Really we only need to be concerned with internal threads to the run-time, right? It's up to apps to restart threads in the children as necessary. > Plus, as long as each module takes reponsibility for itself, > it shouldn't be so hard. > That is, I don't think the case is all that strong for providing the "forkall" > function that is in the diffs I sent. Yes, I think that is overkill. > I'll do more testing and send out diffs "later". > It could be as long as a week, I have a bunch of things to do. > Or maybe just a day. :) > > > I tested a form of this fix + cvsup + user threads and it appears > that cvsup can do one small thing and thereby work either way, > without knowledge as to the way. > > > That is, it can shutdown the dispatcher "directly", instead of queuing to it. > I'm not even sure the dispatcher thread is really needed. > > > As I said, I don't believe the application can handle this all itself. > Any library that uses pthreads is obligated to play into the general > mechanism. Generally the internals are not exposed for the application > to do it. > > Treating Modula-3 as a closed system in which nobody uses > anything outside of or "below" m3core/libm3 I don't think is practical. > > > "applications", arguably defines as "processes", are often composed > of fairly disparate bodies of code that don't necessarily expose much > to each other. For example, they might each create their own worker > threads and not expose them. > This is what pthread_atfork is for. > Actually Tony, you gave me the lead here. :) ;-) > > > There is a problem that m3posixthreads (user) and pthreads semantics > diverage, and..I haven't been able to think of an way to fix that. > > > Well, it is actually easy to make "forkall" the default and only behavior actually, > on user and pthreads (but not NT). > > > But I don't see a way to make "fork1" the behavior for user threads. > You can associate a pid with each thread, and notice when the pid changes, > but I don't know which one thread you would keep, and you wouldn't > necessarily be able to register pthread_atfork handlers, unless maybe > user threads were used only on platforms that actually have pthreads.. > > > And still there's no good way to it on NT I expect. > > > - Jay > > From: jay.krell at cornell.edu > To: dragisha at m3w.org > Date: Thu, 18 Mar 2010 10:09:38 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > Dragisha, Really? The server? With the -C flag and/or -f? > > Evidence is very very very very very good as to what is going on here. > It is all related to "fork1" vs. "forkall". > fork1 is the overwhelming usual behavior (Solaris has an option), > and basically *any* library that uses pthreads is obligated to use > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > The application cannot do things, unless > the library exposes its internal locks. The Posix spec for > pthread_atfork explains the problem. > > Consider C code that links in Modula-3 code. > C code is fairly free to use the fork + do work model. It isn't very common, > but it does exist, and people have gone to extra length to keep it viable. > bash and ccache I believe both use this model. > > > Granted, if you had your own kernel thread implementation, it might > have addressed this. > > > I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions. > The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork. > I want to still try out the "bigger" change, but with the simpler lock/unlock. > And test on Solaris and Darwin. > > > I think for SuspendOthers to not work is not surprising. > The threads can still hold a few locks even if suspended, I'm pretty sure. > The point is to fork in a "controlled" fashion -- all locks held, and in > the right order, so that both parent and child can release them. > > > Taking "all" the locks in Prepare is more reliable. > > > I do wonder if really "all" locks need to be taken. > I only take the static ones declared in PThreadThread.c. > > - Jay > > > > Subject: Re: [M3devel] cvsup > > From: dragisha at m3w.org > > To: jay.krell at cornell.edu > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > Yes, I can. I am building it regularly, from version to version, at > > least few times a year. And it also worked with my ancient kernel thread > > implementation. Was one of stress tests. > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > -- > > Dragi?a Duri? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 18 21:55:06 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 18 Mar 2010 20:55:06 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <1268828983.2906.0.camel@faramir.m3w.org>, , Message-ID: > Name should probably be something more like Thread.AtFork. I disagree. It should be PThreadAtFork, because it only does anything when using pthreads. If you are on a non-pthread system (user threads or Win32), I expect, I think, the function won't have any affect. Granted, we could be using user threads on a system that has pthreads, so there is ambiguity, there is ambiguity either way. Does it mean, hey, thin wrapper over pthread_atfork, and I'll call it even if using user threads? Or does it mean, only call it if using pthreads? > I wonder if we should only fork when the collector is turned off? I don't quite follow. You mean, anyone calling fork() must GC.Disable()? Or LockHeap() Or, take all the thread locks? That last point is part of what I'm saying, since the "prepare" callback would do that. pthread_atfork lets you establish preconditions for fork() "distributed" out around the code, including with access to internals. Without having to "coordinate" with the caller of fork(). > fork + exec is probably relatively safe already. Right, but this change "unnecessarily" alters it. The "prepare" callback would still run. You can't discern fork + noexec vs. fork + exec They start the same. I think we are converging on agreement. Let me do some more tuning (stylistic, diff reduction, not perf) and testing. Diffs later, maybe tonight, maybe in a week, depending on life. > Perhaps a way to disable that -- no way in Posix, we could just > provide a boolean to ourselves. > Not sure I follow. I meant, something like, in m3core/libm3 where we have a fork+exec sequence, set a boolean somewhere to tell all of our "prepare" handlers not to waste their time doing anything. fork + exec need not run the prepare handlers. And I worry that all their lock taking might deadlock..but I suspect that would indicate a bug. ? > recreate the threads > That's tough to do portably! Easy with user threads -- automatically happens. Easy with pthreads -- systems with fork(). Hard/impossible on Win32. > Really we only need to be concerned with internal threads > to the run-time, right? It's up to apps to restart threads > in the children as necessary. I can agree with that. It is kind of just a lazy app that says "hey, please recreate all my threads, I didn't keep track of them, so I don't know what to restart, I know you kept track of all of them, so please do it for me". You should be able to "imagine" what my diff looks like. And maybe ok it without seeing it? Anyway, I definitely want to see Darwin not hang first. - Jay Subject: Re: [M3devel] cvsup From: hosking at cs.purdue.edu Date: Thu, 18 Mar 2010 13:16:03 -0400 CC: dragisha at m3w.org; m3devel at elegosoft.com To: jay.krell at cornell.edu On 18 Mar 2010, at 11:45, Jay K wrote: To repeat myself: > and basically *any* library that uses pthreads is obligated to use > pthread_atfork, be it a C library or the Modula-3 library, etc.. This is the crux of the matter. We are being "bad pthread citizens" by not using pthread_atfork. Granted, we are probably in large company. There is a smaller fix and a larger fix, but it seems very clear we should take one of them. The small fix is: common/Thread.i3: add PThreadAtFork Name should probably be something more like Thread.AtFork. It does nothing on posix/nt. It calls pthread_atfork on pthread. In RTCollector.m3 give a child handler that stores FALSE in the three booleans that indicate the three threads have started. No prepare or parent handler I believe. Though I should check for global locks there. Probably the locks in ThreadPThread suffice? I wonder if we should only fork when the collector is turned off? In ThreadPThread.m3 provide three handlers: prepare: lock "everything", at least the 5 static locks. I might try walking all the threads and locking them too. parent: unlock everything child: unlock everything and reinitialize the globals Note that the atfork handlers run in many more programs than cvsup. They would be used also with fork + exec. fork + exec is probably relatively safe already. Perhaps a way to disable that -- no way in Posix, we could just provide a boolean to ourselves. Not sure I follow. The larger fix would be to provide a common/Thread.i3 function to call fork() *and* recreate all Modula-3 threads in the child process. That is what I sent out. That's tough to do portably! In the second case, you also then need to allow that the caller may or may not use that function, so you still need atfork handlers, but they have to be slightly different, as the diffs I sent show. I believe strongly this is the way to go. It is how one is a "good pthread citizen". Now, granted, I don't think most libraries are. Witness the caveat about fork() in the Darwin manpage and I think even the Posix spec. But pthread_atfork is meant to solve this. I don't suggest a full audit and add atfork handlers everywhere. But m3core we can at least do, and that should satisfy cvsup. Should check libm3 too. Really we only need to be concerned with internal threads to the run-time, right? It's up to apps to restart threads in the children as necessary. Plus, as long as each module takes reponsibility for itself, it shouldn't be so hard. That is, I don't think the case is all that strong for providing the "forkall" function that is in the diffs I sent. Yes, I think that is overkill. I'll do more testing and send out diffs "later". It could be as long as a week, I have a bunch of things to do. Or maybe just a day. :) I tested a form of this fix + cvsup + user threads and it appears that cvsup can do one small thing and thereby work either way, without knowledge as to the way. That is, it can shutdown the dispatcher "directly", instead of queuing to it. I'm not even sure the dispatcher thread is really needed. As I said, I don't believe the application can handle this all itself. Any library that uses pthreads is obligated to play into the general mechanism. Generally the internals are not exposed for the application to do it. Treating Modula-3 as a closed system in which nobody uses anything outside of or "below" m3core/libm3 I don't think is practical. "applications", arguably defines as "processes", are often composed of fairly disparate bodies of code that don't necessarily expose much to each other. For example, they might each create their own worker threads and not expose them. This is what pthread_atfork is for. Actually Tony, you gave me the lead here. :) ;-) There is a problem that m3posixthreads (user) and pthreads semantics diverage, and..I haven't been able to think of an way to fix that. Well, it is actually easy to make "forkall" the default and only behavior actually, on user and pthreads (but not NT). But I don't see a way to make "fork1" the behavior for user threads. You can associate a pid with each thread, and notice when the pid changes, but I don't know which one thread you would keep, and you wouldn't necessarily be able to register pthread_atfork handlers, unless maybe user threads were used only on platforms that actually have pthreads.. And still there's no good way to it on NT I expect. - Jay From: jay.krell at cornell.edu To: dragisha at m3w.org Date: Thu, 18 Mar 2010 10:09:38 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup Dragisha, Really? The server? With the -C flag and/or -f? Evidence is very very very very very good as to what is going on here. It is all related to "fork1" vs. "forkall". fork1 is the overwhelming usual behavior (Solaris has an option), and basically *any* library that uses pthreads is obligated to use pthread_atfork, be it a C library or the Modula-3 library, etc.. The application cannot do things, unless the library exposes its internal locks. The Posix spec for pthread_atfork explains the problem. Consider C code that links in Modula-3 code. C code is fairly free to use the fork + do work model. It isn't very common, but it does exist, and people have gone to extra length to keep it viable. bash and ccache I believe both use this model. Granted, if you had your own kernel thread implementation, it might have addressed this. I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions. The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork. I want to still try out the "bigger" change, but with the simpler lock/unlock. And test on Solaris and Darwin. I think for SuspendOthers to not work is not surprising. The threads can still hold a few locks even if suspended, I'm pretty sure. The point is to fork in a "controlled" fashion -- all locks held, and in the right order, so that both parent and child can release them. Taking "all" the locks in Prepare is more reliable. I do wonder if really "all" locks need to be taken. I only take the static ones declared in PThreadThread.c. - Jay > Subject: Re: [M3devel] cvsup > From: dragisha at m3w.org > To: jay.krell at cornell.edu > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > Yes, I can. I am building it regularly, from version to version, at > least few times a year. And it also worked with my ancient kernel thread > implementation. Was one of stress tests. > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 18 22:33:30 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 18 Mar 2010 17:33:30 -0400 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <1268828983.2906.0.camel@faramir.m3w.org>, , Message-ID: First off, AtFork probably does not belong in the Thread interface. That is part of the language definition and we shouldn't monkey with it. I guess I don't understand the specs of what you are proposing. I can't say that I can immediately imagine your diff. On 18 Mar 2010, at 16:55, Jay K wrote: > > Name should probably be something more like Thread.AtFork. > > > I disagree. > It should be PThreadAtFork, because it only does anything when > using pthreads. If you are on a non-pthread system (user threads or Win32), > I expect, I think, the function won't have any affect. > Granted, we could be using user threads on a system that has pthreads, > so there is ambiguity, there is ambiguity either way. > > > Does it mean, hey, thin wrapper over pthread_atfork, and I'll call it > even if using user threads? Or does it mean, only call it if > using pthreads? > > > > I wonder if we should only fork when the collector is turned off? > > > I don't quite follow. > You mean, anyone calling fork() must GC.Disable()? > Or LockHeap() > Or, take all the thread locks? > That last point is part of what I'm saying, since the "prepare" callback > would do that. > > > pthread_atfork lets you establish preconditions for fork() "distributed" > out around the code, including with access to internals. > Without having to "coordinate" with the caller of fork(). > > > > fork + exec is probably relatively safe already. > > > Right, but this change "unnecessarily" alters it. > The "prepare" callback would still run. > You can't discern fork + noexec vs. fork + exec > They start the same. > > > I think we are converging on agreement. > Let me do some more tuning (stylistic, diff reduction, not perf) and testing. > Diffs later, maybe tonight, maybe in a week, depending on life. > > > > Perhaps a way to disable that -- no way in Posix, we could just > > provide a boolean to ourselves. > > Not sure I follow. > > > I meant, something like, in m3core/libm3 where we have a fork+exec > sequence, set a boolean somewhere to tell all of our "prepare" handlers > not to waste their time doing anything. fork + exec need not > run the prepare handlers. > And I worry that all their lock taking might deadlock..but I suspect > that would indicate a bug. ? > > > > recreate the threads > > That's tough to do portably! > > > Easy with user threads -- automatically happens. > Easy with pthreads -- systems with fork(). > Hard/impossible on Win32. > > > > Really we only need to be concerned with internal threads > > to the run-time, right? It's up to apps to restart threads > > in the children as necessary. > > > I can agree with that. > It is kind of just a lazy app that says "hey, please recreate > all my threads, I didn't keep track of them, so I don't > know what to restart, I know you kept track of all of them, > so please do it for me". > > > You should be able to "imagine" what my diff looks like. > And maybe ok it without seeing it? > > > Anyway, I definitely want to see Darwin not hang first. > > > - Jay > > > > > > > > Subject: Re: [M3devel] cvsup > From: hosking at cs.purdue.edu > Date: Thu, 18 Mar 2010 13:16:03 -0400 > CC: dragisha at m3w.org; m3devel at elegosoft.com > To: jay.krell at cornell.edu > > On 18 Mar 2010, at 11:45, Jay K wrote: > > To repeat myself: > > > and basically *any* library that uses pthreads is obligated to use > > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > > This is the crux of the matter. > We are being "bad pthread citizens" by not using pthread_atfork. > Granted, we are probably in large company. > > > There is a smaller fix and a larger fix, but it seems very clear we > should take one of them. > > > The small fix is: > common/Thread.i3: add PThreadAtFork > > Name should probably be something more like Thread.AtFork. > > It does nothing on posix/nt. > It calls pthread_atfork on pthread. > > > In RTCollector.m3 give a child handler that stores FALSE in the > three booleans that indicate the three threads have started. > No prepare or parent handler I believe. > Though I should check for global locks there. > Probably the locks in ThreadPThread suffice? > > I wonder if we should only fork when the collector is turned off? > > In ThreadPThread.m3 provide three handlers: > prepare: lock "everything", at least the 5 static locks. > I might try walking all the threads and locking them too. > parent: unlock everything > child: unlock everything and reinitialize the globals > > > Note that the atfork handlers run in many more programs than cvsup. > They would be used also with fork + exec. > > fork + exec is probably relatively safe already. > > Perhaps a way to disable that -- no way in Posix, we could just > provide a boolean to ourselves. > > Not sure I follow. > > The larger fix would be to provide a common/Thread.i3 function > to call fork() *and* recreate all Modula-3 threads in the child process. > That is what I sent out. > > That's tough to do portably! > > In the second case, you also then need to allow that the caller > may or may not use that function, so you still need atfork handlers, > but they have to be slightly different, as the diffs I sent show. > > > I believe strongly this is the way to go. > It is how one is a "good pthread citizen". > Now, granted, I don't think most libraries are. > Witness the caveat about fork() in the Darwin manpage and > I think even the Posix spec. > But pthread_atfork is meant to solve this. > > > I don't suggest a full audit and add atfork handlers everywhere. > But m3core we can at least do, and that should satisfy cvsup. > Should check libm3 too. > > Really we only need to be concerned with internal threads to the run-time, right? It's up to apps to restart threads in the children as necessary. > > Plus, as long as each module takes reponsibility for itself, > it shouldn't be so hard. > That is, I don't think the case is all that strong for providing the "forkall" > function that is in the diffs I sent. > > Yes, I think that is overkill. > > I'll do more testing and send out diffs "later". > It could be as long as a week, I have a bunch of things to do. > Or maybe just a day. :) > > > I tested a form of this fix + cvsup + user threads and it appears > that cvsup can do one small thing and thereby work either way, > without knowledge as to the way. > > > That is, it can shutdown the dispatcher "directly", instead of queuing to it. > I'm not even sure the dispatcher thread is really needed. > > > As I said, I don't believe the application can handle this all itself. > Any library that uses pthreads is obligated to play into the general > mechanism. Generally the internals are not exposed for the application > to do it. > > Treating Modula-3 as a closed system in which nobody uses > anything outside of or "below" m3core/libm3 I don't think is practical. > > > "applications", arguably defines as "processes", are often composed > of fairly disparate bodies of code that don't necessarily expose much > to each other. For example, they might each create their own worker > threads and not expose them. > This is what pthread_atfork is for. > Actually Tony, you gave me the lead here. :) > > ;-) > > > > There is a problem that m3posixthreads (user) and pthreads semantics > diverage, and..I haven't been able to think of an way to fix that. > > > Well, it is actually easy to make "forkall" the default and only behavior actually, > on user and pthreads (but not NT). > > > But I don't see a way to make "fork1" the behavior for user threads. > You can associate a pid with each thread, and notice when the pid changes, > but I don't know which one thread you would keep, and you wouldn't > necessarily be able to register pthread_atfork handlers, unless maybe > user threads were used only on platforms that actually have pthreads.. > > > And still there's no good way to it on NT I expect. > > > - Jay > > From: jay.krell at cornell.edu > To: dragisha at m3w.org > Date: Thu, 18 Mar 2010 10:09:38 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > Dragisha, Really? The server? With the -C flag and/or -f? > > Evidence is very very very very very good as to what is going on here. > It is all related to "fork1" vs. "forkall". > fork1 is the overwhelming usual behavior (Solaris has an option), > and basically *any* library that uses pthreads is obligated to use > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > The application cannot do things, unless > the library exposes its internal locks. The Posix spec for > pthread_atfork explains the problem. > > Consider C code that links in Modula-3 code. > C code is fairly free to use the fork + do work model. It isn't very common, > but it does exist, and people have gone to extra length to keep it viable. > bash and ccache I believe both use this model. > > > Granted, if you had your own kernel thread implementation, it might > have addressed this. > > > I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions. > The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork. > I want to still try out the "bigger" change, but with the simpler lock/unlock. > And test on Solaris and Darwin. > > > I think for SuspendOthers to not work is not surprising. > The threads can still hold a few locks even if suspended, I'm pretty sure. > The point is to fork in a "controlled" fashion -- all locks held, and in > the right order, so that both parent and child can release them. > > > Taking "all" the locks in Prepare is more reliable. > > > I do wonder if really "all" locks need to be taken. > I only take the static ones declared in PThreadThread.c. > > - Jay > > > > Subject: Re: [M3devel] cvsup > > From: dragisha at m3w.org > > To: jay.krell at cornell.edu > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > Yes, I can. I am building it regularly, from version to version, at > > least few times a year. And it also worked with my ancient kernel thread > > implementation. Was one of stress tests. > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > -- > > Dragi?a Duri? > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 17:01:03 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements Message-ID: refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 18:21:13 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: Message-ID: diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: atfork2-cvsup.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: atfork2-m3core.txt URL: From jay.krell at cornell.edu Fri Mar 19 19:08:59 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 18:08:59 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , Message-ID: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 19 19:13:32 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 19 Mar 2010 14:13:32 -0400 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , Message-ID: Why not put it into RTProcess? Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > > could go in any of ThreadF, RTOS, RTProcess? > > > Or RTThread. Or almost anywhere. > > > Some typos in the diffs: > in a comment, ThreadF__ should be RTOS__ > function PThreadForkHandler named inconsistently, > should be PThreadAtForkHandler > > > Again, "PThread" appears in these names to indicate > their meaning is not portable. Their existance is, but not their meaning. > We have no way of saying "only call this function for pthreads", > instead as I understand, we provide a portable interface with > drastically varying semantics, such as "do something" vs. "do nothing". > > > Given that Init in ThreadPThread.m3 is private, we could probably > take the stackbase parameter from the caller instead. > > > The call to AtFork should probably follow InitWithStackbase, > in case it fails under low memory, the Die/assert more likely to "work". > > > I'm not completely happy making RTOS public. > So maybe ThreadF? > I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. > But for now they don't exist. > > > Maybe a new interface? ThreadPThread.i3, but is still > present on all platforms, so mostly portable can call it. > > > ThreadPThread.AtFork? > PThread.AtFork? That seems right. > > > - Jay > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 17:21:13 +0000 > Subject: Re: [M3devel] cvsup fix refinements > > diffs attached > > > For now I've removed the Get/SetNextForkWillExec. > I also think truly get/set is right, not inc/dec. > It was confusing enough to interpret and initialize > the value. And it is also not clear what the default > should be. The usual behavor is exec does follow fork. > The safer default if the handlers work ok is to assume > exec will not follow. It is a perf hint or a don't change > how things were hint? Hint to speed up fork+exec or hint > to avoid running the new code that might not work? > > > (For now I've removed the Get/SetNextForkWillExec.) > They'd have to be implemented three times, and > I had trouble defining them. > Obviously Win32/userthreads just return a > hardcoded true, but it is hard to explain > from the caller of Get's point of view. > > Maybe: > SetNextForkWillExec > GetNextForkWillExec > > and then: > PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? > > That is, > someone is who is calling fork and possibly exec, > can immediately tell what these functions mean. > > > But the implementer of an atfork handler has to > do quite a semantic translation I think. > I wrote it backwards the first time. That either > implies it is confusing, or I wasn't thinking. > > > If it really is a problem to run this stuff when exec > will follow, we should know quickly, as building cm3 > is an aggressive user of this code. > > > This version has worked repeatedly on Darwin. > I didn't test user threads but it is structured such > as to make that obviously work. > The cvsup changes are in pthreaad_atfork handlers, > so no change for userthreads. > > > I didn't test on others but confidence is quite high now. > > description of changes at least where they are hard to read: > > m3core: > Init split into Init and InitWithStackBase, > We have to provide the old stack base in the child fork handler > instead of estimating from the address of a local. I don't > know what stack is used, maybe the caller of fork? > i.e. we might have eaten a fair amount of stack and > there could be arbitrary references above the current stack. > > > WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner > > There is a resulting double lock in WeakRefFromRef, every time. > Could instead copy/paste more code around, i.e.: > EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > > > - Jay > > > > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 16:01:03 +0000 > Subject: [M3devel] cvsup fix refinements > > refinements: > > > new public functions in m3core: > > > TYPE PThreadForkHandler = PROCEDURE(); > > <* EXTERNAL RTOS__PThreadAtFork *> > PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; > > > (That's not a change yet.) > > > PROCEDURE SetNextForkWillExec(value: BOOLEAN); > > PROCEDURE GetNextForkWillExec(): BOOLEAN; > > These are only an optimization and should > perhaps be removed? They should be used > under LockHeap/UnlockHeap? which wasn't previously > public. This way, existing fork/exec paths > not affected, though maybe though might as well > ought to be? > > > could go in any of ThreadF, RTOS, RTProcess? > > > Should be called under RTOS.LockHeap? > Therefore I put in RTOS. > Also helps that RTOS is exported from *Thread.m3. > RTOS not previously public. > > > Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) > I implemented them that way: true incs, false decs. > > > Or don't bother with them at all? > > > Furthermore, in RTCollector, instead of claiming the threads > aren't started, if they were started, restart them. > This makes more sense for the weak cleaner to me, and > seems reasonable for the other two. > > > Diffs not yet available. > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 19:15:34 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 18:15:34 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , , Message-ID: RTProcess is fine. Unix is a little wierd because I don't think it should do anything if using userthreads. Unix implies a fairly thin wrapper? - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:13:32 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not put it into RTProcess?Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 19:47:46 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 18:47:46 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , , , , , , Message-ID: How about DoesForkExist, DoesForkInheritAllThreads? From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 18:15:34 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements RTProcess is fine. Unix is a little wierd because I don't think it should do anything if using userthreads. Unix implies a fairly thin wrapper? - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:13:32 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not put it into RTProcess?Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 19 19:47:00 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 19 Mar 2010 14:47:00 -0400 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , Message-ID: <62DD30EE-66EB-444C-84D0-0C1C0B56CD19@cs.purdue.edu> I think the atfork support should go into Unix.i3. It's not necessary for Windows, since Unix.fork is not available there. Also, we should make user and system threading behave the same way. So, for user threads we would destroy all the other threads in the child. This implies that we should have an m3core process fork mechanism. Anyone wanting to fork can then rely on the same behaviour. > Index: src/m3core.h > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/m3core.h,v > retrieving revision 1.8 > diff -u -r1.8 m3core.h > --- src/m3core.h 19 Mar 2010 17:07:58 -0000 1.8 > +++ src/m3core.h 19 Mar 2010 17:08:41 -0000 > @@ -404,6 +404,13 @@ > UINT32 __cdecl Uin__htonl(UINT32 x); > UINT16 __cdecl Uin__htons(UINT16 x); > > +typedef void (*PThreadForkHandler)(void); > + > +INTEGER > +RTOS__PThreadAtFork(PThreadForkHandler prep, > + PThreadForkHandler parent, > + PThreadForkHandler child); > + > #ifdef __cplusplus > } /* extern "C" */ > #endif > Index: src/runtime/common/RTCollector.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/runtime/common/RTCollector.m3,v > retrieving revision 1.88 > diff -u -r1.88 RTCollector.m3 > --- src/runtime/common/RTCollector.m3 15 Dec 2009 19:52:08 -0000 1.88 > +++ src/runtime/common/RTCollector.m3 19 Mar 2010 17:08:42 -0000 > @@ -2033,19 +2033,31 @@ > > VAR startedWeakCleaner := FALSE; > > -PROCEDURE WeakRefFromRef (r: REFANY; p: WeakRefCleanUpProc := NIL): WeakRef = > +PROCEDURE StartWeakCleaner () = > VAR > start := FALSE; > - result: WeakRef; > BEGIN > - <* ASSERT r # NIL *> > TRY > RTOS.LockHeap(); > - (* create a WeakCleaner thread the first time through *) > - IF p # NIL AND NOT startedWeakCleaner THEN > + IF NOT startedWeakCleaner THEN > start := TRUE; > startedWeakCleaner := TRUE; > END; > + FINALLY > + RTOS.UnlockHeap(); > + END; > + IF start THEN > + EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > + END; > + END StartWeakCleaner; > + > +PROCEDURE WeakRefFromRef (r: REFANY; p: WeakRefCleanUpProc := NIL): WeakRef = > + VAR > + result: WeakRef; > + BEGIN > + <* ASSERT r # NIL *> > + TRY > + RTOS.LockHeap(); > (* if necessary, expand weakTable *) > IF weakFree0 = -1 THEN ExpandWeakTable(); END; > IF p # NIL THEN > @@ -2075,9 +2087,8 @@ > FINALLY > RTOS.UnlockHeap(); > END; > - IF start THEN > - EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > - END; > + (* create a WeakCleaner thread the first time through *) > + StartWeakCleaner(); > RETURN result; > END WeakRefFromRef; I would leave WeakRefFromRef unchanged, and simply restart the thread in the child if startedWeakCleaner is TRUE: PROCEDURE AtForkChild() = BEGIN RTOS.LockHeap(); IF startedForeground THEN StartBackgroundCollection(); END; IF startedBackground THEN StartBackgroundCollection(); END; IF startedWeakCleaner THEN StarteWeakCleaner(); END: END AtForkChild; > > @@ -2771,8 +2782,24 @@ > > (*** INITIALIZATION ***) > > +PROCEDURE PThreadForkChild() = > + BEGIN > + IF startedForeground THEN > + StartBackgroundCollection(); > + END; > + IF startedBackground THEN > + StartBackgroundCollection(); > + END; > + IF startedWeakCleaner THEN > + StartWeakCleaner(); > + END; > + END PThreadForkChild; > + > PROCEDURE Init () = > + VAR r: INTEGER; > BEGIN > + r := RTOS.PThreadAtFork((*prepare*)NIL, (*parent*)NIL, PThreadForkChild); > + <* ASSERT r = 0 *> > IF RTParams.IsPresent("paranoidgc") THEN InstallSanityCheck(); END; > IF RTParams.IsPresent("nogc") THEN disableCount := 1; END; > IF RTParams.IsPresent("noincremental") THEN incremental := FALSE; END; > Index: src/runtime/common/RTOS.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/runtime/common/RTOS.i3,v > retrieving revision 1.10 > diff -u -r1.10 RTOS.i3 > --- src/runtime/common/RTOS.i3 8 Sep 2009 05:54:55 -0000 1.10 > +++ src/runtime/common/RTOS.i3 19 Mar 2010 17:08:42 -0000 > @@ -5,7 +5,7 @@ > (*| modified on Sun Feb 21 14:15:00 PST 1993 by jdd *) > (*| modified on Wed Jan 27 22:27:27 PST 1993 by mjordan *) > > -(* "RTOS" is a private interface that provides the low-level, > +(* "RTOS" is a semi-private interface that provides the low-level, > OS-specific memory allocation and shutdown routines. *) Leave this private! > > INTERFACE RTOS; > @@ -40,4 +40,13 @@ > (* Write the "n" bytes beginning at address "a" to the standard > error output file or console. *) > > +(* ThreadF__PThreadAtFork calls pthread_atfork > + * on pthreads platform, otherwise does nothing. > + * Return value is from pthread_atform, 0 for success, > + * always 0 with user threads and Win32 threads. > + *) > +TYPE PThreadForkHandler = PROCEDURE(); > +<* EXTERNAL RTOS__PThreadAtFork *> > +PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; > + Put the pthread_atfork wrapper into Unix.i3. > END RTOS. > Index: src/runtime/common/m3makefile > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/runtime/common/m3makefile,v > retrieving revision 1.27 > diff -u -r1.27 m3makefile > --- src/runtime/common/m3makefile 14 Dec 2009 08:09:48 -0000 1.27 > +++ src/runtime/common/m3makefile 19 Mar 2010 17:08:42 -0000 > @@ -58,7 +58,7 @@ > Interface ("RTSignal") > Interface ("RTStack") > Interface ("RTTypeSRC") > -interface ("RTOS") > +Interface ("RTOS") > > proc FileExists(a) is > return not stale (a, a) > Index: src/thread/POSIX/ThreadPosixC.c > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v > retrieving revision 1.32 > diff -u -r1.32 ThreadPosixC.c > --- src/thread/POSIX/ThreadPosixC.c 16 Dec 2009 12:21:36 -0000 1.32 > +++ src/thread/POSIX/ThreadPosixC.c 19 Mar 2010 17:08:42 -0000 > @@ -279,6 +279,14 @@ > #endif > } > > +INTEGER > +RTOS__PThreadAtFork(PThreadForkHandler prep, > + PThreadForkHandler parent, > + PThreadForkHandler child) > +{ > + return 0; > +} > + > #ifdef __cplusplus > } /* extern "C" */ > #endif > Index: src/thread/PTHREAD/ThreadPThread.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/PTHREAD/ThreadPThread.m3,v > retrieving revision 1.233 > diff -u -r1.233 ThreadPThread.m3 > --- src/thread/PTHREAD/ThreadPThread.m3 18 Mar 2010 11:12:10 -0000 1.233 > +++ src/thread/PTHREAD/ThreadPThread.m3 19 Mar 2010 17:08:42 -0000 > @@ -6,7 +6,7 @@ > Scheduler, SchedulerPosix, RTOS, RTHooks, ThreadPThread; > > IMPORT Cerrno, FloatMode, MutexRep, > - RTCollectorSRC, RTError, RTHeapRep, RTIO, RTParams, > + RTCollectorSRC, RTError, RTHeapRep, RTIO, RTOS, RTParams, > RTPerfTool, RTProcess, ThreadEvent, Time, > Unix, Utime, Word, Usched, > Uerror, Uexec; > @@ -1294,18 +1294,18 @@ > > (*-------------------------------------------------------- Initialization ---*) > > -PROCEDURE Init ()= > +PROCEDURE InitWithStackBase (stackbase: ADDRESS) = > VAR > self: T; > me: Activation; > BEGIN > - InitC(ADR(self)); > + InitC(stackbase); > > me := NEW(Activation, > mutex := pthread_mutex_new(), > cond := pthread_cond_new()); > InitActivations(me); > - me.stackbase := ADR(self); (* not quite accurate but hopefully ok *) > + me.stackbase := stackbase; > IF me.mutex = NIL OR me.cond = NIL THEN > Die(ThisLine(), "Thread initialization failed."); > END; > @@ -1322,8 +1322,48 @@ > IF RTParams.IsPresent("foregroundgc") THEN > RTCollectorSRC.StartForegroundCollection(); > END; > + END InitWithStackBase; > + > +PROCEDURE Init ()= > + VAR r: INTEGER; > + BEGIN > + r := RTOS.PThreadAtFork(PThreadAtForkPrepare, PThreadAtForkParent, PThreadAtForkChild); > + IF r # 0 THEN DieI(ThisLine(), r) END; > + InitWithStackBase(ADR(r)); (* not quite accurate but hopefully ok *) > END Init; > > +VAR locks := ARRAY [0..3] OF pthread_mutex_t{ > + activeMu, slotsMu, initMu, perfMu}; > + > +PROCEDURE PThreadAtForkPrepare() = > + VAR r: int; > + BEGIN > + joinMu.acquire(); > + LockHeap(); > + FOR i := FIRST(locks) TO LAST(locks) DO > + r := pthread_mutex_lock(locks[i]); > + IF r # 0 THEN DieI(ThisLine(), r) END; > + END; > + (* Walk activations and lock all threads? *) > + END PThreadAtForkPrepare; I think there may be deadlock issues here! Would it not be better simply to SuspendOthers() in prepare and ResumeOthers() in parent/child? You would still need to acquire slotsMu/initMu/perfMu before SuspendOthers, and release them after ResumeOthers(). > + > +PROCEDURE PThreadAtForkParent() = > + VAR r: int; > + BEGIN > + FOR i := LAST(locks) TO FIRST(locks) BY -1 DO > + r := pthread_mutex_unlock(locks[i]); > + IF r # 0 THEN DieI(ThisLine(), r) END; > + END; > + UnlockHeap(); > + joinMu.release(); > + END PThreadAtForkParent; > + > +PROCEDURE PThreadAtForkChild() = > + BEGIN > + PThreadAtForkParent(); > + InitWithStackBase(GetActivation().stackbase); > + END PThreadAtForkChild; > + > (*------------------------------------------------------------- collector ---*) > (* These procedures provide synchronization primitives for the allocator > and collector. *) > Index: src/thread/PTHREAD/ThreadPThreadC.c > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/PTHREAD/ThreadPThreadC.c,v > retrieving revision 1.123 > diff -u -r1.123 ThreadPThreadC.c > --- src/thread/PTHREAD/ThreadPThreadC.c 25 Feb 2010 08:31:36 -0000 1.123 > +++ src/thread/PTHREAD/ThreadPThreadC.c 19 Mar 2010 17:08:42 -0000 > @@ -414,6 +414,14 @@ > #endif > } > > +INTEGER > +RTOS__PThreadAtFork(PThreadForkHandler prep, > + PThreadForkHandler parent, > + PThreadForkHandler child) > +{ > + return pthread_atfork(prep, parent, child); > +} > + > #ifdef __cplusplus > } /* extern "C" */ > #endif > Index: src/thread/WIN32/ThreadWin32C.c > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32C.c,v > retrieving revision 1.64 > diff -u -r1.64 ThreadWin32C.c > --- src/thread/WIN32/ThreadWin32C.c 16 Jan 2010 12:44:56 -0000 1.64 > +++ src/thread/WIN32/ThreadWin32C.c 19 Mar 2010 17:08:42 -0000 > @@ -146,6 +146,15 @@ > p(bottom, __getReg(?)); /* bsp? bspstore? try numbers until we find it? */ > #endif > } > + > +/*-------------------------------------------------------------------------*/ > + > +INTEGER > +RTOS__PThreadAtFork(PThreadForkHandler prep, > + PThreadForkHandler parent, > + PThreadForkHandler child) > +{ > + return 0; > } > > /*-------------------------------------------------------------------------*/ On 19 Mar 2010, at 14:13, Tony Hosking wrote: > Why not put it into RTProcess? > Or into Unix? > I want to avoid confusion between Thread.Fork (fork a thread) and process fork. > > On 19 Mar 2010, at 14:08, Jay K wrote: > >> > could go in any of ThreadF, RTOS, RTProcess? >> >> >> Or RTThread. Or almost anywhere. >> >> >> Some typos in the diffs: >> in a comment, ThreadF__ should be RTOS__ >> function PThreadForkHandler named inconsistently, >> should be PThreadAtForkHandler >> >> >> Again, "PThread" appears in these names to indicate >> their meaning is not portable. Their existance is, but not their meaning. >> We have no way of saying "only call this function for pthreads", >> instead as I understand, we provide a portable interface with >> drastically varying semantics, such as "do something" vs. "do nothing". >> >> >> Given that Init in ThreadPThread.m3 is private, we could probably >> take the stackbase parameter from the caller instead. >> >> >> The call to AtFork should probably follow InitWithStackbase, >> in case it fails under low memory, the Die/assert more likely to "work". >> >> >> I'm not completely happy making RTOS public. >> So maybe ThreadF? >> I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. >> But for now they don't exist. >> >> >> Maybe a new interface? ThreadPThread.i3, but is still >> present on all platforms, so mostly portable can call it. >> >> >> ThreadPThread.AtFork? >> PThread.AtFork? That seems right. >> >> >> - Jay >> >> >> From: jay.krell at cornell.edu >> To: m3devel at elegosoft.com >> Date: Fri, 19 Mar 2010 17:21:13 +0000 >> Subject: Re: [M3devel] cvsup fix refinements >> >> diffs attached >> >> >> For now I've removed the Get/SetNextForkWillExec. >> I also think truly get/set is right, not inc/dec. >> It was confusing enough to interpret and initialize >> the value. And it is also not clear what the default >> should be. The usual behavor is exec does follow fork. >> The safer default if the handlers work ok is to assume >> exec will not follow. It is a perf hint or a don't change >> how things were hint? Hint to speed up fork+exec or hint >> to avoid running the new code that might not work? >> >> >> (For now I've removed the Get/SetNextForkWillExec.) >> They'd have to be implemented three times, and >> I had trouble defining them. >> Obviously Win32/userthreads just return a >> hardcoded true, but it is hard to explain >> from the caller of Get's point of view. >> >> Maybe: >> SetNextForkWillExec >> GetNextForkWillExec >> >> and then: >> PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? >> >> That is, >> someone is who is calling fork and possibly exec, >> can immediately tell what these functions mean. >> >> >> But the implementer of an atfork handler has to >> do quite a semantic translation I think. >> I wrote it backwards the first time. That either >> implies it is confusing, or I wasn't thinking. >> >> >> If it really is a problem to run this stuff when exec >> will follow, we should know quickly, as building cm3 >> is an aggressive user of this code. >> >> >> This version has worked repeatedly on Darwin. >> I didn't test user threads but it is structured such >> as to make that obviously work. >> The cvsup changes are in pthreaad_atfork handlers, >> so no change for userthreads. >> >> >> I didn't test on others but confidence is quite high now. >> >> description of changes at least where they are hard to read: >> >> m3core: >> Init split into Init and InitWithStackBase, >> We have to provide the old stack base in the child fork handler >> instead of estimating from the address of a local. I don't >> know what stack is used, maybe the caller of fork? >> i.e. we might have eaten a fair amount of stack and >> there could be arbitrary references above the current stack. >> >> >> WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner >> >> There is a resulting double lock in WeakRefFromRef, every time. >> Could instead copy/paste more code around, i.e.: >> EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); >> >> >> - Jay >> >> >> >> >> >> >> From: jay.krell at cornell.edu >> To: m3devel at elegosoft.com >> Date: Fri, 19 Mar 2010 16:01:03 +0000 >> Subject: [M3devel] cvsup fix refinements >> >> refinements: >> >> >> new public functions in m3core: >> >> >> TYPE PThreadForkHandler = PROCEDURE(); >> >> <* EXTERNAL RTOS__PThreadAtFork *> >> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; >> >> >> (That's not a change yet.) >> >> >> PROCEDURE SetNextForkWillExec(value: BOOLEAN); >> >> PROCEDURE GetNextForkWillExec(): BOOLEAN; >> >> These are only an optimization and should >> perhaps be removed? They should be used >> under LockHeap/UnlockHeap? which wasn't previously >> public. This way, existing fork/exec paths >> not affected, though maybe though might as well >> ought to be? >> >> >> could go in any of ThreadF, RTOS, RTProcess? >> >> >> Should be called under RTOS.LockHeap? >> Therefore I put in RTOS. >> Also helps that RTOS is exported from *Thread.m3. >> RTOS not previously public. >> >> >> Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) >> I implemented them that way: true incs, false decs. >> >> >> Or don't bother with them at all? >> >> >> Furthermore, in RTCollector, instead of claiming the threads >> aren't started, if they were started, restart them. >> This makes more sense for the weak cleaner to me, and >> seems reasonable for the other two. >> >> >> Diffs not yet available. >> >> >> - Jay > From hosking at cs.purdue.edu Fri Mar 19 19:49:14 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 19 Mar 2010 14:49:14 -0400 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , , Message-ID: <7654F263-E59A-4DC0-8225-0F5F3E5BB6C1@cs.purdue.edu> On 19 Mar 2010, at 14:15, Jay K wrote: > RTProcess is fine. > Unix is a little wierd because I don't think it should do anything if using userthreads. I think forking with user threads should have the same behaviour as with system threads: all threads except the forker will die. Another thing. What happens to the dead threads in the child when they get GC'd? Their finaliser will try to invoke CleanThread which might fail because the threads mutex/cond are in a bad state in the child. Is that possible? > Unix implies a fairly thin wrapper? > > - Jay > > > Subject: Re: [M3devel] cvsup fix refinements > From: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 14:13:32 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not put it into RTProcess? > Or into Unix? > I want to avoid confusion between Thread.Fork (fork a thread) and process fork. > > On 19 Mar 2010, at 14:08, Jay K wrote: > > > could go in any of ThreadF, RTOS, RTProcess? > > > Or RTThread. Or almost anywhere. > > > Some typos in the diffs: > in a comment, ThreadF__ should be RTOS__ > function PThreadForkHandler named inconsistently, > should be PThreadAtForkHandler > > > Again, "PThread" appears in these names to indicate > their meaning is not portable. Their existance is, but not their meaning. > We have no way of saying "only call this function for pthreads", > instead as I understand, we provide a portable interface with > drastically varying semantics, such as "do something" vs. "do nothing". > > > Given that Init in ThreadPThread.m3 is private, we could probably > take the stackbase parameter from the caller instead. > > > The call to AtFork should probably follow InitWithStackbase, > in case it fails under low memory, the Die/assert more likely to "work". > > > I'm not completely happy making RTOS public. > So maybe ThreadF? > I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. > But for now they don't exist. > > > Maybe a new interface? ThreadPThread.i3, but is still > present on all platforms, so mostly portable can call it. > > > ThreadPThread.AtFork? > PThread.AtFork? That seems right. > > > - Jay > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 17:21:13 +0000 > Subject: Re: [M3devel] cvsup fix refinements > > diffs attached > > > For now I've removed the Get/SetNextForkWillExec. > I also think truly get/set is right, not inc/dec. > It was confusing enough to interpret and initialize > the value. And it is also not clear what the default > should be. The usual behavor is exec does follow fork. > The safer default if the handlers work ok is to assume > exec will not follow. It is a perf hint or a don't change > how things were hint? Hint to speed up fork+exec or hint > to avoid running the new code that might not work? > > > (For now I've removed the Get/SetNextForkWillExec.) > They'd have to be implemented three times, and > I had trouble defining them. > Obviously Win32/userthreads just return a > hardcoded true, but it is hard to explain > from the caller of Get's point of view. > > Maybe: > SetNextForkWillExec > GetNextForkWillExec > > and then: > PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? > > That is, > someone is who is calling fork and possibly exec, > can immediately tell what these functions mean. > > > But the implementer of an atfork handler has to > do quite a semantic translation I think. > I wrote it backwards the first time. That either > implies it is confusing, or I wasn't thinking. > > > If it really is a problem to run this stuff when exec > will follow, we should know quickly, as building cm3 > is an aggressive user of this code. > > > This version has worked repeatedly on Darwin. > I didn't test user threads but it is structured such > as to make that obviously work. > The cvsup changes are in pthreaad_atfork handlers, > so no change for userthreads. > > > I didn't test on others but confidence is quite high now. > > description of changes at least where they are hard to read: > > m3core: > Init split into Init and InitWithStackBase, > We have to provide the old stack base in the child fork handler > instead of estimating from the address of a local. I don't > know what stack is used, maybe the caller of fork? > i.e. we might have eaten a fair amount of stack and > there could be arbitrary references above the current stack. > > > WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner > > There is a resulting double lock in WeakRefFromRef, every time. > Could instead copy/paste more code around, i.e.: > EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > > > - Jay > > > > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 16:01:03 +0000 > Subject: [M3devel] cvsup fix refinements > > refinements: > > > new public functions in m3core: > > > TYPE PThreadForkHandler = PROCEDURE(); > > <* EXTERNAL RTOS__PThreadAtFork *> > PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; > > > (That's not a change yet.) > > > PROCEDURE SetNextForkWillExec(value: BOOLEAN); > > PROCEDURE GetNextForkWillExec(): BOOLEAN; > > These are only an optimization and should > perhaps be removed? They should be used > under LockHeap/UnlockHeap? which wasn't previously > public. This way, existing fork/exec paths > not affected, though maybe though might as well > ought to be? > > > could go in any of ThreadF, RTOS, RTProcess? > > > Should be called under RTOS.LockHeap? > Therefore I put in RTOS. > Also helps that RTOS is exported from *Thread.m3. > RTOS not previously public. > > > Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) > I implemented them that way: true incs, false decs. > > > Or don't bother with them at all? > > > Furthermore, in RTCollector, instead of claiming the threads > aren't started, if they were started, restart them. > This makes more sense for the weak cleaner to me, and > seems reasonable for the other two. > > > Diffs not yet available. > > > - Jay > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 19 19:50:01 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 19 Mar 2010 14:50:01 -0400 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , , , , , , Message-ID: <83C24180-1290-4F1D-A777-D283475A543F@cs.purdue.edu> The way fork is defined on modern platforms I don't think the child should ever inherit all the threads. On 19 Mar 2010, at 14:47, Jay K wrote: > How about DoesForkExist, DoesForkInheritAllThreads? > > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 18:15:34 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup fix refinements > > RTProcess is fine. > Unix is a little wierd because I don't think it should do anything if using userthreads. > Unix implies a fairly thin wrapper? > > - Jay > > > Subject: Re: [M3devel] cvsup fix refinements > From: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 14:13:32 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not put it into RTProcess? > Or into Unix? > I want to avoid confusion between Thread.Fork (fork a thread) and process fork. > > On 19 Mar 2010, at 14:08, Jay K wrote: > > > could go in any of ThreadF, RTOS, RTProcess? > > > Or RTThread. Or almost anywhere. > > > Some typos in the diffs: > in a comment, ThreadF__ should be RTOS__ > function PThreadForkHandler named inconsistently, > should be PThreadAtForkHandler > > > Again, "PThread" appears in these names to indicate > their meaning is not portable. Their existance is, but not their meaning. > We have no way of saying "only call this function for pthreads", > instead as I understand, we provide a portable interface with > drastically varying semantics, such as "do something" vs. "do nothing". > > > Given that Init in ThreadPThread.m3 is private, we could probably > take the stackbase parameter from the caller instead. > > > The call to AtFork should probably follow InitWithStackbase, > in case it fails under low memory, the Die/assert more likely to "work". > > > I'm not completely happy making RTOS public. > So maybe ThreadF? > I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. > But for now they don't exist. > > > Maybe a new interface? ThreadPThread.i3, but is still > present on all platforms, so mostly portable can call it. > > > ThreadPThread.AtFork? > PThread.AtFork? That seems right. > > > - Jay > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 17:21:13 +0000 > Subject: Re: [M3devel] cvsup fix refinements > > diffs attached > > > For now I've removed the Get/SetNextForkWillExec. > I also think truly get/set is right, not inc/dec. > It was confusing enough to interpret and initialize > the value. And it is also not clear what the default > should be. The usual behavor is exec does follow fork. > The safer default if the handlers work ok is to assume > exec will not follow. It is a perf hint or a don't change > how things were hint? Hint to speed up fork+exec or hint > to avoid running the new code that might not work? > > > (For now I've removed the Get/SetNextForkWillExec.) > They'd have to be implemented three times, and > I had trouble defining them. > Obviously Win32/userthreads just return a > hardcoded true, but it is hard to explain > from the caller of Get's point of view. > > Maybe: > SetNextForkWillExec > GetNextForkWillExec > > and then: > PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? > > That is, > someone is who is calling fork and possibly exec, > can immediately tell what these functions mean. > > > But the implementer of an atfork handler has to > do quite a semantic translation I think. > I wrote it backwards the first time. That either > implies it is confusing, or I wasn't thinking. > > > If it really is a problem to run this stuff when exec > will follow, we should know quickly, as building cm3 > is an aggressive user of this code. > > > This version has worked repeatedly on Darwin. > I didn't test user threads but it is structured such > as to make that obviously work. > The cvsup changes are in pthreaad_atfork handlers, > so no change for userthreads. > > > I didn't test on others but confidence is quite high now. > > description of changes at least where they are hard to read: > > m3core: > Init split into Init and InitWithStackBase, > We have to provide the old stack base in the child fork handler > instead of estimating from the address of a local. I don't > know what stack is used, maybe the caller of fork? > i.e. we might have eaten a fair amount of stack and > there could be arbitrary references above the current stack. > > > WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner > > There is a resulting double lock in WeakRefFromRef, every time. > Could instead copy/paste more code around, i.e.: > EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > > > - Jay > > > > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 16:01:03 +0000 > Subject: [M3devel] cvsup fix refinements > > refinements: > > > new public functions in m3core: > > > TYPE PThreadForkHandler = PROCEDURE(); > > <* EXTERNAL RTOS__PThreadAtFork *> > PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; > > > (That's not a change yet.) > > > PROCEDURE SetNextForkWillExec(value: BOOLEAN); > > PROCEDURE GetNextForkWillExec(): BOOLEAN; > > These are only an optimization and should > perhaps be removed? They should be used > under LockHeap/UnlockHeap? which wasn't previously > public. This way, existing fork/exec paths > not affected, though maybe though might as well > ought to be? > > > could go in any of ThreadF, RTOS, RTProcess? > > > Should be called under RTOS.LockHeap? > Therefore I put in RTOS. > Also helps that RTOS is exported from *Thread.m3. > RTOS not previously public. > > > Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) > I implemented them that way: true incs, false decs. > > > Or don't bother with them at all? > > > Furthermore, in RTCollector, instead of claiming the threads > aren't started, if they were started, restart them. > This makes more sense for the weak cleaner to me, and > seems reasonable for the other two. > > > Diffs not yet available. > > > - Jay > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 20:56:04 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 19:56:04 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: <83C24180-1290-4F1D-A777-D283475A543F@cs.purdue.edu> References: , , , , , , , , , , , , <83C24180-1290-4F1D-A777-D283475A543F@cs.purdue.edu> Message-ID: Agreed. Details later. From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:50:01 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements The way fork is defined on modern platforms I don't think the child should ever inherit all the threads. On 19 Mar 2010, at 14:47, Jay K wrote:How about DoesForkExist, DoesForkInheritAllThreads? From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 18:15:34 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements RTProcess is fine. Unix is a little wierd because I don't think it should do anything if using userthreads. Unix implies a fairly thin wrapper? - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:13:32 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not put it into RTProcess?Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 19 21:17:34 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 19 Mar 2010 16:17:34 -0400 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , , , , , , , , , , , , <83C24180-1290-4F1D-A777-D283475A543F@cs.purdue.edu> Message-ID: <70C491E7-6699-4556-A92C-763B22CBD6DA@cs.purdue.edu> I meant for both user and system threads. On 19 Mar 2010, at 15:56, Jay K wrote: > Agreed. Details later. > > From: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 14:50:01 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup fix refinements > > The way fork is defined on modern platforms I don't think the child should ever inherit all the threads. > > On 19 Mar 2010, at 14:47, Jay K wrote: > > How about DoesForkExist, DoesForkInheritAllThreads? > > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 18:15:34 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup fix refinements > > RTProcess is fine. > Unix is a little wierd because I don't think it should do anything if using userthreads. > Unix implies a fairly thin wrapper? > > - Jay > > > Subject: Re: [M3devel] cvsup fix refinements > From: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 14:13:32 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not put it into RTProcess? > Or into Unix? > I want to avoid confusion between Thread.Fork (fork a thread) and process fork. > > On 19 Mar 2010, at 14:08, Jay K wrote: > > > could go in any of ThreadF, RTOS, RTProcess? > > > Or RTThread. Or almost anywhere. > > > Some typos in the diffs: > in a comment, ThreadF__ should be RTOS__ > function PThreadForkHandler named inconsistently, > should be PThreadAtForkHandler > > > Again, "PThread" appears in these names to indicate > their meaning is not portable. Their existance is, but not their meaning. > We have no way of saying "only call this function for pthreads", > instead as I understand, we provide a portable interface with > drastically varying semantics, such as "do something" vs. "do nothing". > > > Given that Init in ThreadPThread.m3 is private, we could probably > take the stackbase parameter from the caller instead. > > > The call to AtFork should probably follow InitWithStackbase, > in case it fails under low memory, the Die/assert more likely to "work". > > > I'm not completely happy making RTOS public. > So maybe ThreadF? > I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. > But for now they don't exist. > > > Maybe a new interface? ThreadPThread.i3, but is still > present on all platforms, so mostly portable can call it. > > > ThreadPThread.AtFork? > PThread.AtFork? That seems right. > > > - Jay > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 17:21:13 +0000 > Subject: Re: [M3devel] cvsup fix refinements > > diffs attached > > > For now I've removed the Get/SetNextForkWillExec. > I also think truly get/set is right, not inc/dec. > It was confusing enough to interpret and initialize > the value. And it is also not clear what the default > should be. The usual behavor is exec does follow fork. > The safer default if the handlers work ok is to assume > exec will not follow. It is a perf hint or a don't change > how things were hint? Hint to speed up fork+exec or hint > to avoid running the new code that might not work? > > > (For now I've removed the Get/SetNextForkWillExec.) > They'd have to be implemented three times, and > I had trouble defining them. > Obviously Win32/userthreads just return a > hardcoded true, but it is hard to explain > from the caller of Get's point of view. > > Maybe: > SetNextForkWillExec > GetNextForkWillExec > > and then: > PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? > > That is, > someone is who is calling fork and possibly exec, > can immediately tell what these functions mean. > > > But the implementer of an atfork handler has to > do quite a semantic translation I think. > I wrote it backwards the first time. That either > implies it is confusing, or I wasn't thinking. > > > If it really is a problem to run this stuff when exec > will follow, we should know quickly, as building cm3 > is an aggressive user of this code. > > > This version has worked repeatedly on Darwin. > I didn't test user threads but it is structured such > as to make that obviously work. > The cvsup changes are in pthreaad_atfork handlers, > so no change for userthreads. > > > I didn't test on others but confidence is quite high now. > > description of changes at least where they are hard to read: > > m3core: > Init split into Init and InitWithStackBase, > We have to provide the old stack base in the child fork handler > instead of estimating from the address of a local. I don't > know what stack is used, maybe the caller of fork? > i.e. we might have eaten a fair amount of stack and > there could be arbitrary references above the current stack. > > > WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner > > There is a resulting double lock in WeakRefFromRef, every time. > Could instead copy/paste more code around, i.e.: > EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > > > - Jay > > > > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 16:01:03 +0000 > Subject: [M3devel] cvsup fix refinements > > refinements: > > > new public functions in m3core: > > > TYPE PThreadForkHandler = PROCEDURE(); > > <* EXTERNAL RTOS__PThreadAtFork *> > PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; > > > (That's not a change yet.) > > > PROCEDURE SetNextForkWillExec(value: BOOLEAN); > > PROCEDURE GetNextForkWillExec(): BOOLEAN; > > These are only an optimization and should > perhaps be removed? They should be used > under LockHeap/UnlockHeap? which wasn't previously > public. This way, existing fork/exec paths > not affected, though maybe though might as well > ought to be? > > > could go in any of ThreadF, RTOS, RTProcess? > > > Should be called under RTOS.LockHeap? > Therefore I put in RTOS. > Also helps that RTOS is exported from *Thread.m3. > RTOS not previously public. > > > Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) > I implemented them that way: true incs, false decs. > > > Or don't bother with them at all? > > > Furthermore, in RTCollector, instead of claiming the threads > aren't started, if they were started, restart them. > This makes more sense for the weak cleaner to me, and > seems reasonable for the other two. > > > Diffs not yet available. > > > - Jay > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 23:49:15 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 22:49:15 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: <70C491E7-6699-4556-A92C-763B22CBD6DA@cs.purdue.edu> References: , , , , , , , , , , , , <83C24180-1290-4F1D-A777-D283475A543F@cs.purdue.edu> , <70C491E7-6699-4556-A92C-763B22CBD6DA@cs.purdue.edu> Message-ID: I was going to ask: Do you mean pthreads or Modula-3 threads? How you think it is or how you think it should be? Modula-3 user threads are inherited. That is what this function would say. At least they are currently. However, as you assert, I've also wondered many times to myself the past few days: Should Modula-3 userthreads and pthreads act the same way? Always? Optionally? What stopped me was I couldn't convince myself it is trivial to implement. I was thinking, like, you'd associate a pid with each thread. If getpid doesn't match thread.pid, the thread is from before a fork. I then worried that you'd build up all the data about threads "forever", across multiple forks. However that is probably false. As well, if you provide RTProcess.Fork() as the replacement for fork(), it affords more power or at least ease of implementation. I guess where you end is like: RTProcess.Fork() does nothing on Windows in fact is <* EXTERNAL *> and fails to link most likely You could have the convention of returning ENOSYS but I say no. Or it might be a NIL pointer, following the convention you know about, have used several times. just calls fork() on pthreads on user threads, described later RTProcess.AtFork() (maybe RegisterAtFork or RegisterForkCallbacks?) does nothing on Windows (but exists and links, usable by portable code) might be NIL on pthreads just calls pthread_atfork on user threads, described later And then RTProcess.Fork(), RTProcess.AtFork on user threads work together to provide basically the same meaning as fork() + pthread_fork() has on pthreads. Easy to imagine how to code it actually. AtFork maintains an array of function pointers. Fork() calls them and fork in the proscribed order. The array is inherited across processes (like pthread_atfork()). Only the calling thread survives, like fork(). I didn't realize it at first, but killing of all other threads is pretty easy. You just reinitialize like I did for pthreads. Then the next time the timer expires, it doesn't find them to be scheduled. I also strongly suspect that in libm3 where fork+exec is called, some of what it does -- in particular turning off the timer -- would be moved to RTProcess.Fork(). Or perhaps that is in ThreadPosix.m3's AtForkPrepare. Either way, the general idea is that fork is easier to use, be it fork + exec, or even fork + do work. You replace it with RTProcess.Fork(). As well however, fork + exec on pthreads continues to work. You don't have to call RTProcess.Fork(). However RTProcess.Fork() is needed to provide matching semantics in user threads if you don't exec after fork. Geez this is a mouthful. I think it is somewhat important to dwell on -- what semantics change, which external libraries just work and don't have to be wrapped. if you are doing fork + exec, you can keep doing the same, I think if you don't care about user threads, and want them all to be inherited, you can keep doing the same, I think No working code should have an altered meaning. cvsupd on pthreads doesn't count as working code, of course. In fact no fork + do work code on pthreads probably worked. My assertion that "we should use pthread_atfork and be good pthread citizens" remains true and respected by these changes. I'll code this up over the next few days/week. I'm not sure anything ends up in Unix, could all end up in RTProcessC.c or such. If RTProcess.i3 isn't public, it will be. Or we can introduce a new interface. This all sounds about right? Notice that, like my second set of diffs, existing pthreads fork + exec paths will be "significantly" affected -- taking all the locks in the prepare call. It should work. - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 16:17:34 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu I meant for both user and system threads. On 19 Mar 2010, at 15:56, Jay K wrote: Agreed. Details later. From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:50:01 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements The way fork is defined on modern platforms I don't think the child should ever inherit all the threads. On 19 Mar 2010, at 14:47, Jay K wrote: How about DoesForkExist, DoesForkInheritAllThreads? From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 18:15:34 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements RTProcess is fine. Unix is a little wierd because I don't think it should do anything if using userthreads. Unix implies a fairly thin wrapper? - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:13:32 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not put it into RTProcess? Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 23:58:08 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 22:58:08 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: <7654F263-E59A-4DC0-8225-0F5F3E5BB6C1@cs.purdue.edu> References: , , , , , , , <7654F263-E59A-4DC0-8225-0F5F3E5BB6C1@cs.purdue.edu> Message-ID: > What happens to the dead threads in the child when they get GC'd? > Their finaliser will try to invoke CleanThread which might fail because > the threads mutex/cond are in a bad state in the child. Is that possible? This is what pthread_atfork is for. I'm not sure you don't realize that, or are pointing out that I wasn't handling everything. You see the question mark in the ThreadPThread.AtForkPrepare in the diff I sent? That covers this..and I think indeed it needs work: I think for this reason Thread.AtForkPrepare should walk all the threads and take their locks, and then child will release them all, and we can go ahead and cleanup them up right away as well (in the child), we might otherwise lose all references to them, since they were in thread locals. I think AtForkPrepare needs to broadly speaking, acquire *all* locks. Not just some. The diffs I sent only take some. I will include this change in next set of diffs -- the ones that will also change user threads to not inherit threads after RTProcess.Fork(). I suspect this is a difficult task in some code bases. And I'm not sure I suggest clean this up throughout cm3. However we can at least address RTCollector.m3 and ThreadPThread.m3 and get cvsup working. And then we could be like Apple and the docs could just say "fork not followed by exec not guaranteed to work much". ? Also makes me wonder..we should provide wrapped up fork+exec, eh, we already do. It is in libm3. It should probably be in m3core, but I don't think that difference matters too much. - Jay From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:49:14 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements On 19 Mar 2010, at 14:15, Jay K wrote: RTProcess is fine. Unix is a little wierd because I don't think it should do anything if using userthreads. I think forking with user threads should have the same behaviour as with system threads: all threads except the forker will die. Another thing. What happens to the dead threads in the child when they get GC'd? Their finaliser will try to invoke CleanThread which might fail because the threads mutex/cond are in a bad state in the child. Is that possible? Unix implies a fairly thin wrapper? - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:13:32 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not put it into RTProcess? Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 20 04:17:07 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 20 Mar 2010 03:17:07 +0000 Subject: [M3devel] cvsup In-Reply-To: <1268916971.2586.3.camel@faramir.m3w.org> References: ,,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <1268828983.2906.0.camel@faramir.m3w.org>, , <1268916971.2586.3.camel@faramir.m3w.org> Message-ID: Dead as in was working, but no longer? Please elaborate if not working. What does it do? What platform? Can you please debug it? I been doing mostly RTIO.PutText/Flush anyway. I tried several "debuggers": gdb, m3gdb, dbx, couldn't get much out of any. - Jay > Subject: RE: [M3devel] cvsup > From: dragisha at m3w.org > To: jay.krell at cornell.edu > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > Date: Thu, 18 Mar 2010 13:56:11 +0100 > > Client, it's what I am using. And it's dead as we speak. > > Last worked before I pulled/built RC4 > > > On Thu, 2010-03-18 at 10:09 +0000, Jay K wrote: > > Dragisha, Really? The server? With the -C flag and/or -f? > > > > Evidence is very very very very very good as to what is going on here. > > It is all related to "fork1" vs. "forkall". > > fork1 is the overwhelming usual behavior (Solaris has an option), > > and basically *any* library that uses pthreads is obligated to use > > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > > > The application cannot do things, unless > > the library exposes its internal locks. The Posix spec for > > pthread_atfork explains the problem. > > > > Consider C code that links in Modula-3 code. > > C code is fairly free to use the fork + do work model. It isn't very > > common, > > but it does exist, and people have gone to extra length to keep it > > viable. > > bash and ccache I believe both use this model. > > > > > > Granted, if you had your own kernel thread implementation, it might > > have addressed this. > > > > > > I have a version now that has served over 1000 requests, similar to > > what I sent, but a bit reduced. The main change was I just called > > pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead > > of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just > > Linux. This version also doesn't expose the ForkProcessAndAllThreads > > functions. > > The client has to restart its threads, or in the cvsupd case, don't > > depend on the dispatcher thread to survive fork. > > I want to still try out the "bigger" change, but with the simpler > > lock/unlock. > > And test on Solaris and Darwin. > > > > > > I think for SuspendOthers to not work is not surprising. > > The threads can still hold a few locks even if suspended, I'm pretty > > sure. > > The point is to fork in a "controlled" fashion -- all locks held, and > > in > > the right order, so that both parent and child can release them. > > > > > > Taking "all" the locks in Prepare is more reliable. > > > > > > I do wonder if really "all" locks need to be taken. > > I only take the static ones declared in PThreadThread.c. > > > > - Jay > > > > > > > Subject: Re: [M3devel] cvsup > > > From: dragisha at m3w.org > > > To: jay.krell at cornell.edu > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > > > Yes, I can. I am building it regularly, from version to version, at > > > least few times a year. And it also worked with my ancient kernel > > thread > > > implementation. Was one of stress tests. > > > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > > > (Can anyone claim to have seen cvsup server work with kernel > > threads?) > > > -- > > > Dragi?a Duri? > > > > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 20 09:48:05 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 20 Mar 2010 08:48:05 +0000 Subject: [M3devel] pthread_atfork with user threads? Message-ID: Or should the userthreads version implement itself and require user to call RTProcess.Fork()? You know, we have this odd but ok/useful state: Even though we support userthreads, every system has pthreads. I still have to verify that all systems have pthread_atfork, but I assume so. Also funny thing btw, pthreads, userthreads, win32 threads, the code will be highly shared. They will call RegisterForkHandlers. The fork handlers will be same or similar. Specifically in ThreadPThread.c and ThreadPosix.c -- reinitialize. For ThreadPosix that should achieve "don't switch to any preexisting thread", which is equivalent to "kill all other threads". implied question: Call it AtFork or RegisterForkHandlers or ? RTProcessC.c typedef void (*ForkHandler)(void); #ifndef _WIN32 #include #include #include #endif /* NOTE: Even userthreads now depends * on availability of pthreads. * This can be fixed if need be. */ int RTProcess__RegisterForkHandlers( ForkHandler prepare, ForkHandler parent, ForkHandler child) { #ifdef _WIN32 return 0; #else while (1) { int i = pthread_atfork(prepare, parent, child); if (i != EAGAIN) return i; sleep(0); } #endif } -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 20 18:53:49 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 20 Mar 2010 17:53:49 +0000 Subject: [M3devel] atomics AMD64_DARWIN failures? Message-ID: new source -> compiling AtomicBoolean.m3 ../AMD64_DARWIN/AtomicBoolean.m3 => ../src/atomic/Atomic.mg: In function 'AtomicBoolean__Load': ../AMD64_DARWIN/AtomicBoolean.m3 => ../src/atomic/Atomic.mg:0: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicBoolean.mc new source -> compiling AtomicChar.m3 ../AMD64_DARWIN/AtomicChar.m3 => ../src/atomic/Atomic.mg: In function 'AtomicChar__Load': ../AMD64_DARWIN/AtomicChar.m3 => ../src/atomic/Atomic.mg:0: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicChar.mc new source -> compiling AtomicInteger.m3 ../AMD64_DARWIN/AtomicInteger.m3 => ../src/atomic/Atomic.mg: In function 'AtomicInteger__Store': ../AMD64_DARWIN/AtomicInteger.m3 => ../src/atomic/Atomic.mg:21: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicInteger.mc new source -> compiling AtomicLongint.m3 ../AMD64_DARWIN/AtomicLongint.m3 => ../src/atomic/Atomic.mg: In function 'AtomicLongint__Store': ../AMD64_DARWIN/AtomicLongint.m3 => ../src/atomic/Atomic.mg:21: internal compiler error: Bus error Please submit a full bug report, I'll look into it..first just rebuild from scratch. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 20 19:08:12 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 20 Mar 2010 18:08:12 +0000 Subject: [M3devel] atomics AMD64_DARWIN failures? In-Reply-To: References: Message-ID: Sorry, nevermind, new checkout, new build, works fine. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Sat, 20 Mar 2010 17:53:49 +0000 Subject: [M3devel] atomics AMD64_DARWIN failures? new source -> compiling AtomicBoolean.m3 ../AMD64_DARWIN/AtomicBoolean.m3 => ../src/atomic/Atomic.mg: In function 'AtomicBoolean__Load': ../AMD64_DARWIN/AtomicBoolean.m3 => ../src/atomic/Atomic.mg:0: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicBoolean.mc new source -> compiling AtomicChar.m3 ../AMD64_DARWIN/AtomicChar.m3 => ../src/atomic/Atomic.mg: In function 'AtomicChar__Load': ../AMD64_DARWIN/AtomicChar.m3 => ../src/atomic/Atomic.mg:0: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicChar.mc new source -> compiling AtomicInteger.m3 ../AMD64_DARWIN/AtomicInteger.m3 => ../src/atomic/Atomic.mg: In function 'AtomicInteger__Store': ../AMD64_DARWIN/AtomicInteger.m3 => ../src/atomic/Atomic.mg:21: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicInteger.mc new source -> compiling AtomicLongint.m3 ../AMD64_DARWIN/AtomicLongint.m3 => ../src/atomic/Atomic.mg: In function 'AtomicLongint__Store': ../AMD64_DARWIN/AtomicLongint.m3 => ../src/atomic/Atomic.mg:21: internal compiler error: Bus error Please submit a full bug report, I'll look into it..first just rebuild from scratch. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 20 21:52:52 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 20 Mar 2010 20:52:52 +0000 Subject: [M3devel] cvsup/fork fix Message-ID: attached is looking pretty good cvsup seems to work on Darwin, kernel threads or user threads I do see: 2010.03.20 13:47:08 PDT [62878]: -0 [0Kin+0Kout] ChannelMux protocol error: Expected EOF, didn't get it but it seems any combination does that. I could try again with no diffs and userthreads. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: atfork4-cvsup.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: atfork4-m3core.txt URL: From jay.krell at cornell.edu Sun Mar 21 02:34:56 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 21 Mar 2010 01:34:56 +0000 Subject: [M3devel] cvsup/fork fix Message-ID: > 2010.03.20 13:47:08 PDT [62878]: -0 [0Kin+0Kout] ChannelMux protocol error: Expected EOF, didn't get it I also see this *occasionally* with user threads and no changes, but very consistently with my changes. I'll probably look into it further. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: cvsup/fork fix Date: Sat, 20 Mar 2010 20:52:52 +0000 attached is looking pretty good cvsup seems to work on Darwin, kernel threads or user threads I do see: 2010.03.20 13:47:08 PDT [62878]: -0 [0Kin+0Kout] ChannelMux protocol error: Expected EOF, didn't get it but it seems any combination does that. I could try again with no diffs and userthreads. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 21 02:40:11 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 21 Mar 2010 01:40:11 +0000 Subject: [M3devel] recent commits Message-ID: commit mail is temporarily down (or is it just me?), so: http://www.modula3.com/cm3/ChangeLog-2010 2010-03-20 20:40 jkrell * m3-libs/m3core/src/runtime/common/RTProcessC.c: new file, empty to start 2010-03-20 07:59 jkrell * m3-libs/m3core/src/Csupport/Common/hand.c: implement set_gt(a,b) as set_lt(b,a) implement set_ge(a,b) as set_le(b,a) Really the frontend or backend should make this transform. 2010-03-20 07:56 jkrell * m3-libs/m3core/src/Csupport/Common/test_hand.c: add some tests for set_lt, le, gt, le; I find this code a bit hard to understand ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 21 08:19:08 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 21 Mar 2010 08:19:08 +0100 Subject: [M3devel] cvsup In-Reply-To: References: ,,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> ,, ,,<1268828983.2906.0.camel@faramir.m3w.org> , ,<1268916971.2586.3.camel@faramir.m3w.org> Message-ID: <1269155948.6829.5.camel@faramir.m3w.org> cvsup client was working for me all the time until latest build - RC4 pulled from local CVS (mirrored by cvsup) on Feb 21 2010. It also worked 5 yrs ago when I used it with my NPTL implementation. What little I understood glimpsing over this discussion, it affects cvsup server? Or is my situation typical? I'll try m3gdb it. Thanks On Sat, 2010-03-20 at 03:17 +0000, Jay K wrote: > Dead as in was working, but no longer? > Please elaborate if not working. What does it do? > What platform? > Can you please debug it? > I been doing mostly RTIO.PutText/Flush anyway. > I tried several "debuggers": gdb, m3gdb, dbx, couldn't get much out of > any. > > - Jay > > > Subject: RE: [M3devel] cvsup > > From: dragisha at m3w.org > > To: jay.krell at cornell.edu > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > Date: Thu, 18 Mar 2010 13:56:11 +0100 > > > > Client, it's what I am using. And it's dead as we speak. > > > > Last worked before I pulled/built RC4 > > > > > > On Thu, 2010-03-18 at 10:09 +0000, Jay K wrote: > > > Dragisha, Really? The server? With the -C flag and/or > -f? > > > > > > Evidence is very very very very very good as to what is going on > here. > > > It is all related to "fork1" vs. "forkall". > > > fork1 is the overwhelming usual behavior (Solaris has an option), > > > and basically *any* library that uses pthreads is obligated to use > > > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > > > > > The application cannot do things, unless > > > the library exposes its internal locks. The Posix spec for > > > pthread_atfork explains the problem. > > > > > > Consider C code that links in Modula-3 code. > > > C code is fairly free to use the fork + do work model. It isn't > very > > > common, > > > but it does exist, and people have gone to extra length to keep it > > > viable. > > > bash and ccache I believe both use this model. > > > > > > > > > Granted, if you had your own kernel thread implementation, it > might > > > have addressed this. > > > > > > > > > I have a version now that has served over 1000 requests, similar > to > > > what I sent, but a bit reduced. The main change was I just called > > > pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, > instead > > > of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just > > > Linux. This version also doesn't expose the > ForkProcessAndAllThreads > > > functions. > > > The client has to restart its threads, or in the cvsupd case, > don't > > > depend on the dispatcher thread to survive fork. > > > I want to still try out the "bigger" change, but with the simpler > > > lock/unlock. > > > And test on Solaris and Darwin. > > > > > > > > > I think for SuspendOthers to not work is not surprising. > > > The threads can still hold a few locks even if suspended, I'm > pretty > > > sure. > > > The point is to fork in a "controlled" fashion -- all locks held, > and > > > in > > > the right order, so that both parent and child can release them. > > > > > > > > > Taking "all" the locks in Prepare is more reliable. > > > > > > > > > I do wonder if really "all" locks need to be taken. > > > I only take the static ones declared in PThreadThread.c. > > > > > > - Jay > > > > > > > > > > Subject: Re: [M3devel] cvsup > > > > From: dragisha at m3w.org > > > > To: jay.krell at cornell.edu > > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > > > > > Yes, I can. I am building it regularly, from version to version, > at > > > > least few times a year. And it also worked with my ancient > kernel > > > thread > > > > implementation. Was one of stress tests. > > > > > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > > > > > (Can anyone claim to have seen cvsup server work with kernel > > > threads?) > > > > -- > > > > Dragi?a Duri? > > > > > > -- > > Dragi?a Duri? > > -- Dragi?a Duri? From jay.krell at cornell.edu Sun Mar 21 09:54:50 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 21 Mar 2010 08:54:50 +0000 Subject: [M3devel] cvsup In-Reply-To: <1269155948.6829.5.camel@faramir.m3w.org> References: ,,,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , , , <1268828983.2906.0.camel@faramir.m3w.org>, , , , <1268916971.2586.3.camel@faramir.m3w.org>, , <1269155948.6829.5.camel@faramir.m3w.org> Message-ID: Dragisha, it's still not clear to me if client is still working for not or not, on which platform, and if it doesn't work, what does it do? (Well, NPTL implies Linux.) Yes, mostly we are talking about server here. The server would always hang with kernel threads. But I'm going to look at client more due to this "EOF expected" warning I'm getting. > Or is my situation typical? What is your situation? We have very few reports of any kind, positive or negative. "Dead" could mean you stopped using it. But if it means it stopped working, please, what does it to? If it hangs, put in PutText+Flush calls to find out where it gets to before it hangs? See if it works with user threads (edit m3core/src/thread.quake)? - Jay > Subject: RE: [M3devel] cvsup > From: dragisha at m3w.org > To: jay.krell at cornell.edu > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > Date: Sun, 21 Mar 2010 08:19:08 +0100 > > cvsup client was working for me all the time until latest build - RC4 > pulled from local CVS (mirrored by cvsup) on Feb 21 2010. > > It also worked 5 yrs ago when I used it with my NPTL implementation. > > What little I understood glimpsing over this discussion, it affects > cvsup server? Or is my situation typical? > > I'll try m3gdb it. > > Thanks > > On Sat, 2010-03-20 at 03:17 +0000, Jay K wrote: > > Dead as in was working, but no longer? > > Please elaborate if not working. What does it do? > > What platform? > > Can you please debug it? > > I been doing mostly RTIO.PutText/Flush anyway. > > I tried several "debuggers": gdb, m3gdb, dbx, couldn't get much out of > > any. > > > > - Jay > > > > > Subject: RE: [M3devel] cvsup > > > From: dragisha at m3w.org > > > To: jay.krell at cornell.edu > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > Date: Thu, 18 Mar 2010 13:56:11 +0100 > > > > > > Client, it's what I am using. And it's dead as we speak. > > > > > > Last worked before I pulled/built RC4 > > > > > > > > > On Thu, 2010-03-18 at 10:09 +0000, Jay K wrote: > > > > Dragisha, Really? The server? With the -C flag and/or > > -f? > > > > > > > > Evidence is very very very very very good as to what is going on > > here. > > > > It is all related to "fork1" vs. "forkall". > > > > fork1 is the overwhelming usual behavior (Solaris has an option), > > > > and basically *any* library that uses pthreads is obligated to use > > > > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > > > > > > > The application cannot do things, unless > > > > the library exposes its internal locks. The Posix spec for > > > > pthread_atfork explains the problem. > > > > > > > > Consider C code that links in Modula-3 code. > > > > C code is fairly free to use the fork + do work model. It isn't > > very > > > > common, > > > > but it does exist, and people have gone to extra length to keep it > > > > viable. > > > > bash and ccache I believe both use this model. > > > > > > > > > > > > Granted, if you had your own kernel thread implementation, it > > might > > > > have addressed this. > > > > > > > > > > > > I have a version now that has served over 1000 requests, similar > > to > > > > what I sent, but a bit reduced. The main change was I just called > > > > pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, > > instead > > > > of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just > > > > Linux. This version also doesn't expose the > > ForkProcessAndAllThreads > > > > functions. > > > > The client has to restart its threads, or in the cvsupd case, > > don't > > > > depend on the dispatcher thread to survive fork. > > > > I want to still try out the "bigger" change, but with the simpler > > > > lock/unlock. > > > > And test on Solaris and Darwin. > > > > > > > > > > > > I think for SuspendOthers to not work is not surprising. > > > > The threads can still hold a few locks even if suspended, I'm > > pretty > > > > sure. > > > > The point is to fork in a "controlled" fashion -- all locks held, > > and > > > > in > > > > the right order, so that both parent and child can release them. > > > > > > > > > > > > Taking "all" the locks in Prepare is more reliable. > > > > > > > > > > > > I do wonder if really "all" locks need to be taken. > > > > I only take the static ones declared in PThreadThread.c. > > > > > > > > - Jay > > > > > > > > > > > > > Subject: Re: [M3devel] cvsup > > > > > From: dragisha at m3w.org > > > > > To: jay.krell at cornell.edu > > > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > > > > > > > Yes, I can. I am building it regularly, from version to version, > > at > > > > > least few times a year. And it also worked with my ancient > > kernel > > > > thread > > > > > implementation. Was one of stress tests. > > > > > > > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > > > > > > > (Can anyone claim to have seen cvsup server work with kernel > > > > threads?) > > > > > -- > > > > > Dragi?a Duri? > > > > > > > > -- > > > Dragi?a Duri? > > > > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 21 10:33:11 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 21 Mar 2010 10:33:11 +0100 Subject: [M3devel] cvsup In-Reply-To: References: ,,,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> ,,, ,,,<1268828983.2906.0.camel@faramir.m3w.org> ,, ,,<1268916971.2586.3.camel@faramir.m3w.org> , ,<1269155948.6829.5.camel@faramir.m3w.org> Message-ID: <1269163991.6829.15.camel@faramir.m3w.org> Few versions ago (May or July, when I last tried) cvsup started to behave ill - it just stops giving any output while memory usage is skyrocketing. With that version, I've synced "incrementally". When it locks, I kill it, then start over. With latest version, one from Feb 21, this does not help anymore. It just stops working (I've not observed if it leaks memory as former version). Half an hour ago I've reinstalled rpm's for one of this former versions I've had on my system. cvsup again sort-of-works. At least it finishes syncing after few restarts. This is what I get with -v: faramir:dragisha/pts/6: ~% cvsup -v CVSup client, non-GUI version Copyright 1996-2003 John D. Polstra Software version: 2009-08-03 16:02:04 Protocol version: 17.0 Operating system: LINUXLIBC6 It looks, for me, like client is culprit. Someone maybe can follow this even further back to find when and where. I don't think kernel threads are so guilty of this behaviour. I will build RC cvs head now. On Sun, 2010-03-21 at 08:54 +0000, Jay K wrote: > Dragisha, it's still not clear to me if client is still working for > not or not, > on which platform, and if it doesn't work, what does it do? > (Well, NPTL implies Linux.) > Yes, mostly we are talking about server here. > The server would always hang with kernel threads. > But I'm going to look at client more due to this "EOF expected" > warning I'm getting. > > > > Or is my situation typical? > > > What is your situation? > We have very few reports of any kind, positive or negative. > > > "Dead" could mean you stopped using it. > But if it means it stopped working, please, what does it to? > If it hangs, put in PutText+Flush calls to find out where > it gets to before it hangs? > See if it works with user threads (edit m3core/src/thread.quake)? > > > - Jay > > > Subject: RE: [M3devel] cvsup > > From: dragisha at m3w.org > > To: jay.krell at cornell.edu > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > Date: Sun, 21 Mar 2010 08:19:08 +0100 > > > > cvsup client was working for me all the time until latest build - > RC4 > > pulled from local CVS (mirrored by cvsup) on Feb 21 2010. > > > > It also worked 5 yrs ago when I used it with my NPTL implementation. > > > > What little I understood glimpsing over this discussion, it affects > > cvsup server? Or is my situation typical? > > > > I'll try m3gdb it. > > > > Thanks > > > > On Sat, 2010-03-20 at 03:17 +0000, Jay K wrote: > > > Dead as in was working, but no longer? > > > Please elaborate if not working. What does it do? > > > What platform? > > > Can you please debug it? > > > I been doing mostly RTIO.PutText/Flush anyway. > > > I tried several "debuggers": gdb, m3gdb, dbx, couldn't get much > out of > > > any. > > > > > > - Jay > > > > > > > Subject: RE: [M3devel] cvsup > > > > From: dragisha at m3w.org > > > > To: jay.krell at cornell.edu > > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > > Date: Thu, 18 Mar 2010 13:56:11 +0100 > > > > > > > > Client, it's what I am using. And it's dead as we speak. > > > > > > > > Last worked before I pulled/built RC4 > > > > > > > > > > > > On Thu, 2010-03-18 at 10:09 +0000, Jay K wrote: > > > > > Dragisha, Really? The server? With the -C flag and/or > > > -f? > > > > > > > > > > Evidence is very very very very very good as to what is going > on > > > here. > > > > > It is all related to "fork1" vs. "forkall". > > > > > fork1 is the overwhelming usual behavior (Solaris has an > option), > > > > > and basically *any* library that uses pthreads is obligated to > use > > > > > pthread_atfork, be it a C library or the Modula-3 library, > etc.. > > > > > > > > > > The application cannot do things, unless > > > > > the library exposes its internal locks. The Posix spec for > > > > > pthread_atfork explains the problem. > > > > > > > > > > Consider C code that links in Modula-3 code. > > > > > C code is fairly free to use the fork + do work model. It > isn't > > > very > > > > > common, > > > > > but it does exist, and people have gone to extra length to > keep it > > > > > viable. > > > > > bash and ccache I believe both use this model. > > > > > > > > > > > > > > > Granted, if you had your own kernel thread implementation, it > > > might > > > > > have addressed this. > > > > > > > > > > > > > > > I have a version now that has served over 1000 requests, > similar > > > to > > > > > what I sent, but a bit reduced. The main change was I just > called > > > > > pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, > > > instead > > > > > of using SuspendOthers. Haven't tested on Solaris/Darwin yet, > just > > > > > Linux. This version also doesn't expose the > > > ForkProcessAndAllThreads > > > > > functions. > > > > > The client has to restart its threads, or in the cvsupd case, > > > don't > > > > > depend on the dispatcher thread to survive fork. > > > > > I want to still try out the "bigger" change, but with the > simpler > > > > > lock/unlock. > > > > > And test on Solaris and Darwin. > > > > > > > > > > > > > > > I think for SuspendOthers to not work is not surprising. > > > > > The threads can still hold a few locks even if suspended, I'm > > > pretty > > > > > sure. > > > > > The point is to fork in a "controlled" fashion -- all locks > held, > > > and > > > > > in > > > > > the right order, so that both parent and child can release > them. > > > > > > > > > > > > > > > Taking "all" the locks in Prepare is more reliable. > > > > > > > > > > > > > > > I do wonder if really "all" locks need to be taken. > > > > > I only take the static ones declared in PThreadThread.c. > > > > > > > > > > - Jay > > > > > > > > > > > > > > > > Subject: Re: [M3devel] cvsup > > > > > > From: dragisha at m3w.org > > > > > > To: jay.krell at cornell.edu > > > > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > > > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > > > > > > > > > Yes, I can. I am building it regularly, from version to > version, > > > at > > > > > > least few times a year. And it also worked with my ancient > > > kernel > > > > > thread > > > > > > implementation. Was one of stress tests. > > > > > > > > > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > > > > > > > > > (Can anyone claim to have seen cvsup server work with > kernel > > > > > threads?) > > > > > > -- > > > > > > Dragi?a Duri? > > > > > > > > > > -- > > > > Dragi?a Duri? > > > > > > -- > > Dragi?a Duri? > > -- Dragi?a Duri? From dragisha at m3w.org Sun Mar 21 18:27:01 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 21 Mar 2010 18:27:01 +0100 Subject: [M3devel] cvsup (fix) state? Message-ID: <1269192421.6829.17.camel@faramir.m3w.org> I've just pulled latest cm3 RC and built up to cvsup... Started client, and it stalls - again - sometimes during process. Is this expected? Fix talk is still talk only? -- Dragi?a Duri? From Highjinks at gmx.com Tue Mar 23 21:02:32 2010 From: Highjinks at gmx.com (Chris) Date: Tue, 23 Mar 2010 21:02:32 +0100 (CET) Subject: [M3devel] Moving ahead with Modula3. Making my code safe. Message-ID: <20100323160650.74e96922.Highjinks@gmx.com> Alright, I'm finally comfortable enough with Modula3 to start doing some serious work with it. However, I'm now in the process of learning how to do things the Modula3 way. Or probably more correctly, writing things that are easily managed by other Modula3 developers. What is the "proper" Modula3 way of taking an unsafe interface to an external library and making it "safe" for the purposes of my fellow Modula3 developers? I know how I would it with Ada and GNAT GPL; but even though the languages are similiar, the environments are quite a bit different. The same thing goes for pretty much everything else. I really want to nail down my libraries and make them as bulletproof as possible. I'd like to make everything as exception free as possible. This means very few uses of the "RAISES" keyword. If a properly defined function returns a predefined error, I dont consider that exceptional. I typically consider any undefined behavior to be exceptional. Also, does Modula3 provide any facilities for Synchronous threads and/or State Machines/Automata, or do these all have to be coded manually? One last question, in regards to threads...how do I get access to Atomic ops? Typically I like to prefetch a dataset into the processors data cache and do most of my operations there. I can doubletime it on a multicore processor, assuming I dont have to use Mutexes and such. Any tips, pointers, or articles you might recommend? -- Chris From wagner at elegosoft.com Wed Mar 24 17:14:54 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Wed, 24 Mar 2010 17:14:54 +0100 Subject: [M3devel] Fwd: m3 Message-ID: <20100324171454.u9gw7n1bz4kcwcs8@mail.elegosoft.com> Any takers? All detail information missing so far :-( Olaf ----- Forwarded message from sunglee726 at gmail.com ----- Date: Wed, 24 Mar 2010 06:01:40 -0500 From: Sung Lee Reply-To: Sung Lee Subject: m3 To: m3-support at elego.de When I compile a modula-3 program and run it, I get an error saying the MSVCR90.dll was not found. What should I do? ----- End forwarded message ----- -------------- next part -------------- When I compile a modula-3 program and run it, I get an error saying the MSVCR90.dll was not found. What should I do? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 27 04:35:17 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 27 Mar 2010 03:35:17 +0000 Subject: [M3devel] interface UserThread? Message-ID: Should we offer, say, interface UserThread? One could say IMPORT UserThread AS Thread. It's pretty easy, on Posix. Except there isn't just Thread.T exposed by the thread library. There is e.g. Scheduler, SchedulerPosix. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Sat Mar 27 16:38:40 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Sat, 27 Mar 2010 16:38:40 +0100 Subject: [M3devel] interface UserThread? In-Reply-To: References: Message-ID: <20100327163840.81nebtotwkws0488@mail.elegosoft.com> Quoting Jay K : > Should we offer, say, interface UserThread? > One could say IMPORT UserThread AS Thread. > > It's pretty easy, on Posix. > Except there isn't just Thread.T exposed by the thread library. > There is e.g. Scheduler, SchedulerPosix. You'd like to mix those interfaces? Let's go one step back. What we would actually like to have is a simple switch for system and user threads, for example by just linking against another library. This isn't possible in the current implementation, because _everything_ would have to be recompiled, IIRC. I'm not sure about the reason anymore though. Why can't we have all thread-specific calls pass through one library interface? Does the compiler need to produce different code for some calls? Is it just too inefficient for some functions? If we cannot achieve something simpler that would not need changing the code, then I'd consider a new interface for user threads an option. I don't really like it, because Thread is a standard interface of the language, and we should be very careful to change anything there. At least we should agree that it's worth the efforts. Any opinions? Olaf From wagner at elegosoft.com Sat Mar 27 17:05:53 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Sat, 27 Mar 2010 17:05:53 +0100 Subject: [M3devel] Moving ahead with Modula3. Making my code safe. In-Reply-To: <20100323160650.74e96922.Highjinks@gmx.com> References: <20100323160650.74e96922.Highjinks@gmx.com> Message-ID: <20100327170553.wn2dv2t3448880cs@mail.elegosoft.com> Quoting Chris : > Alright, I'm finally comfortable enough with Modula3 to start doing > some serious work with it. That sounds promising :-) > However, I'm now in the process of learning how to do things the > Modula3 way. Or probably more correctly, writing things that are > easily managed by other Modula3 developers. > > What is the "proper" Modula3 way of taking an unsafe interface to an > external library and making it "safe" for the purposes of my fellow > Modula3 developers? I know how I would it with Ada and GNAT GPL; > but even though the languages are similiar, the environments are > quite a bit different. I don't think you can really make external C and C++ code safe in the sense that Modula-3 code is safe. What you can do is to make sure that the types are mapped correctly, there are no memory leaks and memory overwrites, that all parameters actually are in the expected range, and the returned values are interpreted correctly in the enclosing M3 code. Interfaces to C or C++ are never safe. You can do your best to make the module that uses them safe, i.e. there should be no undetected runtime error or unnoticed side-effect by calling methods of your module. > The same thing goes for pretty much everything else. I really want > to nail down my libraries and make them as bulletproof as possible. Yes, that's the general idea :-) > I'd like to make everything as exception free as possible. This > means very few uses of the "RAISES" keyword. If a properly defined > function returns a predefined error, I dont consider that > exceptional. I typically consider any undefined behavior to be > exceptional. This depends. Exceptions, as the name suggests, should be used for exceptional situations that occur rather infrequently. They may also impose some performance penalty depending on their implementation. The usual, error-free code path should not involve any exceptions. > Also, does Modula3 provide any facilities for Synchronous threads > and/or State Machines/Automata, or do these all have to be coded > manually? Offhand, I don't know of any modules for coroutines or finite state machines. Regarding lexical analysis, you will surely find some FSM-based scanner generators in the compiler tools (caltech-parser). > One last question, in regards to threads...how do I get access to > Atomic ops? Typically I like to prefetch a dataset into the > processors data cache and do most of my operations there. I can > doubletime it on a multicore processor, assuming I dont have to use > Mutexes and such. Atomic operations on processor level are or course very platform specific, and are not exposed in the standard libraries. The standard way to synchronize resource access and make certain operations atomic from an external view is to use mutexes, which are defined in the Thread interface. If this is too high-level or inefficient for certain purposes, Antony Hosking has recently added some generic atomics support in m3core/src/atomic/Atomic.ig. I'm pretty sure it is only in the current head though, and don't know if it's really already implemented on all target platforms. > Any tips, pointers, or articles you might recommend? The language definition as well as a tutorial and a bibliography are online, though I'm afraid that several of the links to external resources are outdated. You surely have browsed through the documentation section of http://www.opencm3.net/? Hope this helps, Olaf From wagner at elegosoft.com Sat Mar 27 17:28:52 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Sat, 27 Mar 2010 17:28:52 +0100 Subject: [M3devel] m3 In-Reply-To: <91819bfb1003240401h6ee7d8fdr70cd68c174cc5d9@mail.gmail.com> References: <91819bfb1003240401h6ee7d8fdr70cd68c174cc5d9@mail.gmail.com> Message-ID: <20100327172852.z6f981if40ww4sok@mail.elegosoft.com> Quoting Sung Lee : > When I compile a modula-3 program and run it, I get an error saying the > MSVCR90.dll was not found. What should I do? Sorry for the delay, I've been very busy again during the week. I forwarded your help request to the m3devel mailing list, but nobody there seems to have been eager to act on it, too. This could be due to the little information you provide though. What exact version of Modula-3 are you using? What commands did you issue, and what C compiler/linker from Microsoft have you installed? It may simply be that the dll in question is not in your path, but I'm not very familiar with Windows systems, so this is just a guess. If you provide more details, others may be able to help. You may also want to create a trac ticket: https://projects.elego.de/cm3/newticket Be sure to fill out all the fields with as much information as possible. Best regards, Olaf From wagner at elegosoft.com Sat Mar 27 18:46:59 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Sat, 27 Mar 2010 18:46:59 +0100 Subject: [M3devel] cvsup/fork fix In-Reply-To: References: Message-ID: <20100327184659.4ipyw1tdicsko0sk@mail.elegosoft.com> Quoting Jay K : > attached is looking pretty good > cvsup seems to work on Darwin, kernel threads or user threads > > I do see: > 2010.03.20 13:47:08 PDT [62878]: -0 [0Kin+0Kout] ChannelMux protocol > error: Expected EOF, didn't get it > > but it seems any combination does that. > I could try again with no diffs and userthreads. What's the status of the fork/threads fixes needed to make CVSup work again? Any chance to get something into the release branch in order to go ahead with RC5? There was also an issue reported with the client. I tried to run it to update the CM3 repository, and it stalls immediately: % /usr/local/cm3/bin/cvsup -g -L 2 -P m -l LOCK_cvsup cvsupfile.cm3 Parsing supfile "cvsupfile.cm3" Connecting to birch.elego.de Connected to birch.elego.de Server software version: release_DCVS_1_0_3_rc8_at_old_elego_de Negotiating file attribute support Exchanging collection information Establishing multiplexed-mode data connection If I run it in the FreeBSD gdb though, it doesn't hang and works as expected (apart from the unnecessary SetAttrs operations). This is with the latest sources from the release branch. It's not deterministic though; sometimes it gets on with its work. I'll try again to catch the deadlock in the debugger. Any other ideas? Olaf From jay.krell at cornell.edu Sun Mar 28 03:26:54 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 28 Mar 2010 01:26:54 +0000 Subject: [M3devel] cvsup/fork Message-ID: The "unexpected EOF" warnings in cvsup worry me, but I'm not sure I have a chance of figuring them out particularly soon. I fear I'll have to read the code a fair amount and look for race conditions, given the intermittent behavior. The changes in m3core which make it work better are correct seemingly independently. I'm inclined to just go with them, at least the pthread part. The EOF warning shows up in the current version even, with user threads, about 1 in 10 times. I'm not seeing m3commit mails. Just me? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 28 08:51:27 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 28 Mar 2010 06:51:27 +0000 Subject: [M3devel] cvsup/SigHandler.m3? Message-ID: This whole file SigHandler.m3 exists so that Modula-3 signal handlers are run in a special thread, free to use any Modula-3 facility. Is it really needed? Is it needed when using pthreads? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From mand at elegosoft.com Sun Mar 28 09:31:41 2010 From: mand at elegosoft.com (mand at elegosoft.com) Date: Sun, 28 Mar 2010 09:31:41 +0200 Subject: [M3devel] test mail Message-ID: <20100328093141.hbtyulj8oo008skg@mail.elego.de> checking for relay to non-elego domains. From jay.krell at cornell.edu Sun Mar 28 10:44:39 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 28 Mar 2010 08:44:39 +0000 Subject: [M3devel] cvsup/fork Message-ID: Mostly nevermind. I'm not seeing the EOF warnings on Linux/x86. Maybe they were Darwin only. I'll look into that "later", but based on Linux/x86 I'll proceed with the fixes. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: cvsup/fork Date: Sun, 28 Mar 2010 01:26:54 +0000 The "unexpected EOF" warnings in cvsup worry me, but I'm not sure I have a chance of figuring them out particularly soon. I fear I'll have to read the code a fair amount and look for race conditions, given the intermittent behavior. The changes in m3core which make it work better are correct seemingly independently. I'm inclined to just go with them, at least the pthread part. The EOF warning shows up in the current version even, with user threads, about 1 in 10 times. I'm not seeing m3commit mails. Just me? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 28 11:58:04 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 28 Mar 2010 09:58:04 +0000 Subject: [M3devel] userthreads vs. pthreads performance? Message-ID: I understand that userthreads don't scale across multiple processors. But testing out cvsup lately, things are noticably much much faster with user threads. At least cvsup. I haven't watched the compiler and such. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 28 12:15:57 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 28 Mar 2010 12:15:57 +0200 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: Message-ID: <1269771357.3671.7.camel@faramir.m3w.org> CVSup w/ user threads is not faster because user threads are faster - it is faster because it's buggy behaviour manifests only with kernel threads. IMO. Algorithm-wise - at least on Linux NPTL is O(1) decision making time, ie switching threads. User threads are lots of linear list walking. Figure how efficient it is. Process management "science" has gone long way since "founding fathers" did user threads thingy. d On Sun, 2010-03-28 at 09:58 +0000, Jay K wrote: > I understand that userthreads don't scale across multiple processors. > But testing out cvsup lately, things are noticably much much faster > with user threads. At least cvsup. I haven't watched the compiler and > such. > > > - Jay > > > > > -- Dragi?a Duri? From mika at async.async.caltech.edu Sun Mar 28 18:24:24 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Sun, 28 Mar 2010 09:24:24 -0700 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269771357.3671.7.camel@faramir.m3w.org> References: <1269771357.3671.7.camel@faramir.m3w.org> Message-ID: <20100328162424.670F91A20B7@async.async.caltech.edu> In my tests, programs that acquire a lot of locks (even uncontended ones, I think) run much much faster with user threads than with kernel threads. Or at least they run much (several times) faster compiled with an ancient PM3 than with a recent CM3, and I'm pretty sure the threading is the main difference. (Given that the effect only occurs for programs that acquire a lot of locks.) Mika =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: >CVSup w/ user threads is not faster because user threads are faster - it >is faster because it's buggy behaviour manifests only with kernel >threads. IMO. > >Algorithm-wise - at least on Linux NPTL is O(1) decision making time, ie >switching threads. User threads are lots of linear list walking. Figure >how efficient it is. Process management "science" has gone long way >since "founding fathers" did user threads thingy. > >d > >On Sun, 2010-03-28 at 09:58 +0000, Jay K wrote: >> I understand that userthreads don't scale across multiple processors. >> But testing out cvsup lately, things are noticably much much faster >> with user threads. At least cvsup. I haven't watched the compiler and >> such. >> >> >> - Jay >> >> >> >> >> >-- >Dragi??a Duri?? From hosking at cs.purdue.edu Sun Mar 28 18:55:36 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 28 Mar 2010 12:55:36 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <20100328162424.670F91A20B7@async.async.caltech.edu> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> Message-ID: <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> Yes, we need to implement thin locks and biased locks. These have minimal overhead for locks held mostly by just one thread. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 28 Mar 2010, at 12:24, Mika Nystrom wrote: > > In my tests, programs that acquire a lot of locks (even uncontended ones, > I think) run much much faster with user threads than with kernel threads. > Or at least they run much (several times) faster compiled with an ancient > PM3 than with a recent CM3, and I'm pretty sure the threading is the > main difference. (Given that the effect only occurs for programs that > acquire a lot of locks.) > > Mika > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: >> CVSup w/ user threads is not faster because user threads are faster - it >> is faster because it's buggy behaviour manifests only with kernel >> threads. IMO. >> >> Algorithm-wise - at least on Linux NPTL is O(1) decision making time, ie >> switching threads. User threads are lots of linear list walking. Figure >> how efficient it is. Process management "science" has gone long way >> since "founding fathers" did user threads thingy. >> >> d >> >> On Sun, 2010-03-28 at 09:58 +0000, Jay K wrote: >>> I understand that userthreads don't scale across multiple processors. >>> But testing out cvsup lately, things are noticably much much faster >>> with user threads. At least cvsup. I haven't watched the compiler and >>> such. >>> >>> >>> - Jay >>> >>> >>> >>> >>> >> -- >> Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 28 20:32:22 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 28 Mar 2010 20:32:22 +0200 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> Message-ID: <1269801143.3671.17.camel@faramir.m3w.org> Fast glance shows there is no overhead in code? Just passthrough of MUTEX field to pthread_mutex_lock()? And inefficiency comes from there? On Sun, 2010-03-28 at 12:55 -0400, Tony Hosking wrote: > Yes, we need to implement thin locks and biased locks. These have > minimal overhead for locks held mostly by just one thread. -- Dragi?a Duri? From mika at async.async.caltech.edu Sun Mar 28 20:57:40 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Sun, 28 Mar 2010 11:57:40 -0700 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269801143.3671.17.camel@faramir.m3w.org> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> <1269801143.3671.17.camel@faramir.m3w.org> Message-ID: <20100328185740.38AFA1A20B9@async.async.caltech.edu> Yep, sounds right. I was profiling some other thread-using code that slowed down enormously because of pthreads and it turned out the program was spending ~95% of its time in accessing the thread locals via one of the pthread_ functions. (The overhead of entering the kernel.) Mika =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: >Fast glance shows there is no overhead in code? Just passthrough of >MUTEX field to pthread_mutex_lock()? And inefficiency comes from there? > >On Sun, 2010-03-28 at 12:55 -0400, Tony Hosking wrote: >> Yes, we need to implement thin locks and biased locks. These have >> minimal overhead for locks held mostly by just one thread. >-- >Dragi??a Duri?? From dragisha at m3w.org Sun Mar 28 21:02:04 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 28 Mar 2010 21:02:04 +0200 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <20100328185740.38AFA1A20B9@async.async.caltech.edu> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> <1269801143.3671.17.camel@faramir.m3w.org> <20100328185740.38AFA1A20B9@async.async.caltech.edu> Message-ID: <1269802924.3671.18.camel@faramir.m3w.org> Which platform? On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > Yep, sounds right. > > I was profiling some other thread-using code that slowed down > enormously > because of pthreads and it turned out the program was spending ~95% > of its time in accessing the thread locals via one of the pthread_ > functions. > (The overhead of entering the kernel.) -- Dragi?a Duri? From hosking at cs.purdue.edu Sun Mar 28 21:08:12 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 28 Mar 2010 15:08:12 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269801143.3671.17.camel@faramir.m3w.org> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> <1269801143.3671.17.camel@faramir.m3w.org> Message-ID: <13B06EBC-1349-4973-BD2B-62C061509471@cs.purdue.edu> Yep. Call. Atomic ops. All expensive. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 28 Mar 2010, at 14:32, Dragi?a Duri? wrote: > Fast glance shows there is no overhead in code? Just passthrough of > MUTEX field to pthread_mutex_lock()? And inefficiency comes from there? > > On Sun, 2010-03-28 at 12:55 -0400, Tony Hosking wrote: >> Yes, we need to implement thin locks and biased locks. These have >> minimal overhead for locks held mostly by just one thread. > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.async.caltech.edu Sun Mar 28 21:11:16 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Sun, 28 Mar 2010 12:11:16 -0700 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269802924.3671.18.camel@faramir.m3w.org> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> <1269801143.3671.17.camel@faramir.m3w.org> <20100328185740.38AFA1A20B9@async.async.caltech.edu> <1269802924.3671.18.camel@faramir.m3w.org> Message-ID: <20100328191116.CD9C71A20BB@async.async.caltech.edu> Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: >Which platform? > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: >> Yep, sounds right. >> >> I was profiling some other thread-using code that slowed down >> enormously >> because of pthreads and it turned out the program was spending ~95% >> of its time in accessing the thread locals via one of the pthread_ >> functions. >> (The overhead of entering the kernel.) >-- >Dragi??a Duri?? From dragisha at m3w.org Sun Mar 28 21:14:57 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 28 Mar 2010 21:14:57 +0200 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <20100328191116.CD9C71A20BB@async.async.caltech.edu> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> <1269801143.3671.17.camel@faramir.m3w.org> <20100328185740.38AFA1A20B9@async.async.caltech.edu> <1269802924.3671.18.camel@faramir.m3w.org> <20100328191116.CD9C71A20BB@async.async.caltech.edu> Message-ID: <1269803697.3671.19.camel@faramir.m3w.org> I remember reading (long time ago) about how these (FUTEXes) are efficient in LINUX... Can I have your test code to try? On Sun, 2010-03-28 at 12:11 -0700, Mika Nystrom wrote: > Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > >Which platform? > > > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > >> Yep, sounds right. > >> > >> I was profiling some other thread-using code that slowed down > >> enormously > >> because of pthreads and it turned out the program was spending ~95% > >> of its time in accessing the thread locals via one of the pthread_ > >> functions. > >> (The overhead of entering the kernel.) > >-- > >Dragi?a Duri? -- Dragi?a Duri? From mika at async.async.caltech.edu Sun Mar 28 22:33:53 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Sun, 28 Mar 2010 13:33:53 -0700 Subject: [M3devel] Extended Static Checker Message-ID: <20100328203353.8507A1A20BB@async.async.caltech.edu> Hello m3devel, I am curious if the source code for the Modula-3 Extended Static Checker is available anywhere? Or a Linux binary might work for my purposes... I seem to remember they did distribute it at some point... Mika From jay.krell at cornell.edu Sun Mar 28 22:46:01 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 28 Mar 2010 20:46:01 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269803697.3671.19.camel@faramir.m3w.org> References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> Message-ID: O(1) scheduling is not a new idea. Just look at NT and probably Solaris and probably all the other non-free systems (AIX, Irix, HP-UX, Tru64, VMS, etc.) Getting thread locals should not require a kernel call. It doesn't on NT. We can optimize this somewhat on most systems with __thread. I had that in briefly. Entering an uncontended pthread mutex should not be expensive -- at least no kernel call, but granted a call and atomic op. Two calls because of the C layer. But user threads pay for a call too of course. Maybe I should profile some of this.. - Jay > From: dragisha at m3w.org > To: mika at async.async.caltech.edu > Date: Sun, 28 Mar 2010 21:14:57 +0200 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > I remember reading (long time ago) about how these (FUTEXes) are > efficient in LINUX... Can I have your test code to try? > > On Sun, 2010-03-28 at 12:11 -0700, Mika Nystrom wrote: > > Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... > > > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > > >Which platform? > > > > > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > > >> Yep, sounds right. > > >> > > >> I was profiling some other thread-using code that slowed down > > >> enormously > > >> because of pthreads and it turned out the program was spending ~95% > > >> of its time in accessing the thread locals via one of the pthread_ > > >> functions. > > >> (The overhead of entering the kernel.) > > >-- > > >Dragi?a Duri? > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Highjinks at gmx.com Sun Mar 28 23:01:31 2010 From: Highjinks at gmx.com (Chris) Date: Sun, 28 Mar 2010 23:01:31 +0200 (CEST) Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: Message-ID: <20100328170619.5d95e6cd.Highjinks@gmx.com> On Sun, 28 Mar 2010 09:58:04 +0000 Jay K wrote: > > I understand that userthreads don't scale across multiple processors. > > But testing out cvsup lately, things are noticably much much faster > > with user threads. At least cvsup. I haven't watched the compiler and such. > > > > > > - Jay In fact user threads, and better yet, Synchronous Threads, are in fact highly scalable when done right. This has been one point of contention for me regarding the current crop of "mature" programming languages. Everything is being shoehorned into the Asynchronous model. The problem is that the Asynchronous model is staring to break down. With more and more processor cores being added, the need for more and more locks and mutexes and semaphores is growing. Not only does this kill performance, it makes problems like deadlock and race conditions much more difficult to prevent. It used to be that user threads, cooperative threads, and Synchronous threads were seen as bottlenecks. That might have been the case back when we were dealing maybe four processors max. But now were looking, in some cases, at machines with four 4 core processors on a MB; and that number is expected to keep growing. Now Cooperative threads and Synchronous threads are outscaling Asynchronous threads, particularly in multimedia applications. The thing that is really needed now is language support and kernel support for the Synchronous model. Here are some reference links, for those who are curious... The Kusp project, which has support for Synchronous threads in the Linux Kernel. http://www.ittc.ku.edu/kurt/ A Wikipedia link on the Esterel programming language. http://en.wikipedia.org/wiki/Esterel Project COSA, which explores the issue a little more in depth. http://www.rebelscience.org/Cosas/COSA.htm The Timber compiler, which is all about Synchronous Programming. For those who care to experiment. http://timber-lang.org/ I realize the OP is about "user threads", but I think it might be good to open up a discussion about "user threads" and the subtopics therein. Modula might benefit from a formal definition of this stuff. IMHO. Anyone else share my interest? -- Chris From jay.krell at cornell.edu Mon Mar 29 03:59:39 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 29 Mar 2010 01:59:39 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <20100328170619.5d95e6cd.Highjinks@gmx.com> References: , <20100328170619.5d95e6cd.Highjinks@gmx.com> Message-ID: People need to know to reduce cross-thread sharing, to reduce lock contention. It is not impossible. There are various conflicting trends though, granted. Windows has long had fibers, but they are pretty impossible to use -- you need your own locking mechanisms, which interact with your own scheduler. There is now "user mode scheduling" in Windows 7 that is easier to use, and the "concurrent runtime" makes it easier to use, along with helping reduce sharing. You should sort of "map/reduce" sorts of things, where each thread can do some of the work independently and there is one last mop-up or sweep pass to collect the data. Consider sorting large data sets for example. You can break up the data into N pieces, sort of them independently on N threads, and then do one last single threaded pass to merge them. "Cloud computing" pushes people toward models of very coarse grained parallelism -- multiple machines, not even the "old coarse grained" multiple processes on the same machine. I think profiling is maybe in order to understand cvsup performance with user threads vs. kernel threads. Usually I ignore performance but the difference is quite noticable in my limited testing. Personally I've only been convinced that kernel threads are generally best and simplest. There are tradeoffs though. Having one mondo scheduler puts a lot of pressure on that one scheduler to figure out things for everyone, whereas within each program there is probably local information that could provide for better scheduling. Basically, taking locks or waiting on objects (files, events, semaphores) is the only way to communicate to a scheduler. It helps a lot, but maybe is not always all that could be done. - Jay > From: Highjinks at gmx.com > To: m3devel at elegosoft.com > Date: Sun, 28 Mar 2010 23:01:31 +0200 > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > On Sun, 28 Mar 2010 09:58:04 +0000 > Jay K wrote: > > > > > I understand that userthreads don't scale across multiple processors. > > > > But testing out cvsup lately, things are noticably much much faster > > > > with user threads. At least cvsup. I haven't watched the compiler and such. > > > > > > > > > > > > - Jay > > In fact user threads, and better yet, Synchronous Threads, are in fact highly scalable when done right. > > This has been one point of contention for me regarding the current crop of "mature" programming languages. Everything is being shoehorned into the Asynchronous model. > > The problem is that the Asynchronous model is staring to break down. With more and more processor cores being added, the need for more and more locks and mutexes and semaphores is growing. Not only does this kill performance, it makes problems like deadlock and race conditions much more difficult to prevent. > > It used to be that user threads, cooperative threads, and Synchronous threads were seen as bottlenecks. That might have been the case back when we were dealing maybe four processors max. But now were looking, in some cases, at machines with four 4 core processors on a MB; and that number is expected to keep growing. Now Cooperative threads and Synchronous threads are outscaling Asynchronous threads, particularly in multimedia applications. > > The thing that is really needed now is language support and kernel support for the Synchronous model. > > Here are some reference links, for those who are curious... > > The Kusp project, which has support for Synchronous threads in the Linux Kernel. > http://www.ittc.ku.edu/kurt/ > > A Wikipedia link on the Esterel programming language. > http://en.wikipedia.org/wiki/Esterel > > Project COSA, which explores the issue a little more in depth. > http://www.rebelscience.org/Cosas/COSA.htm > > The Timber compiler, which is all about Synchronous Programming. For those who care to experiment. > http://timber-lang.org/ > > I realize the OP is about "user threads", but I think it might be good to open up a discussion about "user threads" and the subtopics therein. Modula might benefit from a formal definition of this stuff. IMHO. > > Anyone else share my interest? > > -- > Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Mar 29 04:06:20 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 29 Mar 2010 02:06:20 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <20100328170619.5d95e6cd.Highjinks@gmx.com> References: , <20100328170619.5d95e6cd.Highjinks@gmx.com> Message-ID: http://www.ittc.ku.edu/kurt/ wow, it implies the Linux kernel isn't preemptible...but I did a bit more research..as of 2.6 it is preemtible, configurable, but I don't know what the default is. I failed to mention Erlang, which also pushes process-level concurrency, and much of it. You still need some of locks in these systems, granted. Like file locks. - Jay From: jay.krell at cornell.edu To: highjinks at gmx.com; m3devel at elegosoft.com Subject: RE: [M3devel] userthreads vs. pthreads performance? Date: Mon, 29 Mar 2010 01:59:39 +0000 People need to know to reduce cross-thread sharing, to reduce lock contention. It is not impossible. There are various conflicting trends though, granted. Windows has long had fibers, but they are pretty impossible to use -- you need your own locking mechanisms, which interact with your own scheduler. There is now "user mode scheduling" in Windows 7 that is easier to use, and the "concurrent runtime" makes it easier to use, along with helping reduce sharing. You should sort of "map/reduce" sorts of things, where each thread can do some of the work independently and there is one last mop-up or sweep pass to collect the data. Consider sorting large data sets for example. You can break up the data into N pieces, sort of them independently on N threads, and then do one last single threaded pass to merge them. "Cloud computing" pushes people toward models of very coarse grained parallelism -- multiple machines, not even the "old coarse grained" multiple processes on the same machine. I think profiling is maybe in order to understand cvsup performance with user threads vs. kernel threads. Usually I ignore performance but the difference is quite noticable in my limited testing. Personally I've only been convinced that kernel threads are generally best and simplest. There are tradeoffs though. Having one mondo scheduler puts a lot of pressure on that one scheduler to figure out things for everyone, whereas within each program there is probably local information that could provide for better scheduling. Basically, taking locks or waiting on objects (files, events, semaphores) is the only way to communicate to a scheduler. It helps a lot, but maybe is not always all that could be done. - Jay > From: Highjinks at gmx.com > To: m3devel at elegosoft.com > Date: Sun, 28 Mar 2010 23:01:31 +0200 > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > On Sun, 28 Mar 2010 09:58:04 +0000 > Jay K wrote: > > > > > I understand that userthreads don't scale across multiple processors. > > > > But testing out cvsup lately, things are noticably much much faster > > > > with user threads. At least cvsup. I haven't watched the compiler and such. > > > > > > > > > > > > - Jay > > In fact user threads, and better yet, Synchronous Threads, are in fact highly scalable when done right. > > This has been one point of contention for me regarding the current crop of "mature" programming languages. Everything is being shoehorned into the Asynchronous model. > > The problem is that the Asynchronous model is staring to break down. With more and more processor cores being added, the need for more and more locks and mutexes and semaphores is growing. Not only does this kill performance, it makes problems like deadlock and race conditions much more difficult to prevent. > > It used to be that user threads, cooperative threads, and Synchronous threads were seen as bottlenecks. That might have been the case back when we were dealing maybe four processors max. But now were looking, in some cases, at machines with four 4 core processors on a MB; and that number is expected to keep growing. Now Cooperative threads and Synchronous threads are outscaling Asynchronous threads, particularly in multimedia applications. > > The thing that is really needed now is language support and kernel support for the Synchronous model. > > Here are some reference links, for those who are curious... > > The Kusp project, which has support for Synchronous threads in the Linux Kernel. > http://www.ittc.ku.edu/kurt/ > > A Wikipedia link on the Esterel programming language. > http://en.wikipedia.org/wiki/Esterel > > Project COSA, which explores the issue a little more in depth. > http://www.rebelscience.org/Cosas/COSA.htm > > The Timber compiler, which is all about Synchronous Programming. For those who care to experiment. > http://timber-lang.org/ > > I realize the OP is about "user threads", but I think it might be good to open up a discussion about "user threads" and the subtopics therein. Modula might benefit from a formal definition of this stuff. IMHO. > > Anyone else share my interest? > > -- > Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Mar 29 05:15:41 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 29 Mar 2010 03:15:41 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269803697.3671.19.camel@faramir.m3w.org> References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> Message-ID: > Getting thread locals should not require a kernel call Indeed, on Linux/x86 it does not, looks pretty ok: 00000380 <__pthread_getspecific>: 380: 55 push %ebp 381: 89 e5 mov %esp,%ebp 383: 8b 55 08 mov 0x8(%ebp),%edx 386: 81 fa ff 03 00 00 cmp $0x3ff,%edx 38c: 76 04 jbe 392 <__pthread_getspecific+0x12> 38e: 5d pop %ebp 38f: 31 c0 xor %eax,%eax 391: c3 ret 392: 89 d0 mov %edx,%eax 394: c1 e8 05 shr $0x5,%eax 397: 8d 0c 85 1c 01 00 00 lea 0x11c(,%eax,4),%ecx 39e: 65 8b 01 mov %gs:(%ecx),%eax 3a1: 85 c0 test %eax,%eax 3a3: 74 e9 je 38e <__pthread_getspecific+0xe> 3a5: 8b 04 d5 00 00 00 00 mov 0x0(,%edx,8),%eax 3ac: 85 c0 test %eax,%eax 3ae: 74 de je 38e <__pthread_getspecific+0xe> 3b0: 65 8b 01 mov %gs:(%ecx),%eax 3b3: 83 e2 1f and $0x1f,%edx 3b6: 8b 04 90 mov (%eax,%edx,4),%eax 3b9: 5d pop %ebp 3ba: c3 ret > Entering an uncontended pthread mutex should not be expensive Linux/x86: 00001020 <__pthread_self>: 1020: 55 push %ebp 1021: 89 e5 mov %esp,%ebp 1023: 65 a1 50 00 00 00 mov %gs:0x50,%eax 1029: 5d pop %ebp 102a: c3 ret 102b: 90 nop 102c: 8d 74 26 00 lea 0x0(%esi),%esi pretty lame, five instructions were only two are needed. 000004f0 <__pthread_mutex_lock>: .. too much to read through..but I think no kernel call.. - Jay From: jay.krell at cornell.edu To: dragisha at m3w.org; mika at async.async.caltech.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] userthreads vs. pthreads performance? Date: Sun, 28 Mar 2010 20:46:01 +0000 O(1) scheduling is not a new idea. Just look at NT and probably Solaris and probably all the other non-free systems (AIX, Irix, HP-UX, Tru64, VMS, etc.) Getting thread locals should not require a kernel call. It doesn't on NT. We can optimize this somewhat on most systems with __thread. I had that in briefly. Entering an uncontended pthread mutex should not be expensive -- at least no kernel call, but granted a call and atomic op. Two calls because of the C layer. But user threads pay for a call too of course. Maybe I should profile some of this.. - Jay > From: dragisha at m3w.org > To: mika at async.async.caltech.edu > Date: Sun, 28 Mar 2010 21:14:57 +0200 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > I remember reading (long time ago) about how these (FUTEXes) are > efficient in LINUX... Can I have your test code to try? > > On Sun, 2010-03-28 at 12:11 -0700, Mika Nystrom wrote: > > Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... > > > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > > >Which platform? > > > > > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > > >> Yep, sounds right. > > >> > > >> I was profiling some other thread-using code that slowed down > > >> enormously > > >> because of pthreads and it turned out the program was spending ~95% > > >> of its time in accessing the thread locals via one of the pthread_ > > >> functions. > > >> (The overhead of entering the kernel.) > > >-- > > >Dragi?a Duri? > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Mar 29 10:24:47 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 29 Mar 2010 08:24:47 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269803697.3671.19.camel@faramir.m3w.org> References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> Message-ID: > We can optimize this somewhat on most systems with __thread. I had that in briefly I did a little bit of testing here. __thread takes a few forms, depending on which of -fPIC and -shared you use. Once you do throw in -fPIC and -shared, I have found __thread to be significantly slower on Solaris/sparc and Linux/powerpc32, slower or a wash on Linux/amd64, and twice as fast as pthread_getspecific on Linux/x86. I doesn't appear supported at all on Darwin, though pthread_getspecific are very fast there (albeit not inlined). I didn't test *BSD. My testing was not very scientific. However -fPIC and/or -shared imply a function call to access __thread variables. That's probably the big factor. Without -fPIC/-shared, there is no function call. If you are going to access them multiple times in a function then there is a probably an optimization to be had -- caching their address. If the variables are larger than a pointer, probably then also an optimization to be had. We could do that first thing for certain -- PushFrame could return the address and PopFrame would be much faster. However another angle here is to eliminate PushFrame/PopFrame, by using libunwind. I think we should look into libunwind for the next release. The thread locals will (mostly) remain but accesses to them greatly decline. We compile everything -fPIC and -shared. Some systems (libtool) compile things once that way and once not, providing pairs of libraries, depending on intended use. I should point out also that userthreads have been greatly deoptimized in the current tree (by me, Tony approved), because they used to inline PushFrame/PopFrame, but they don't any longer. (Historically on NT, __declspec(thread) only worked in code in the .exe or statically loaded by the .exe -- that is, not in .dlls loaded with LoadLibrary. However that limitation was removed in Vista. I expect __declspec(thread) is much faster than TlsGetValue, but I assume we'll support pre-Vista for a while longer so not interesting..) - Jay From: jay.krell at cornell.edu To: dragisha at m3w.org; mika at async.async.caltech.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] userthreads vs. pthreads performance? Date: Mon, 29 Mar 2010 03:15:41 +0000 > Getting thread locals should not require a kernel call Indeed, on Linux/x86 it does not, looks pretty ok: 00000380 <__pthread_getspecific>: 380: 55 push %ebp 381: 89 e5 mov %esp,%ebp 383: 8b 55 08 mov 0x8(%ebp),%edx 386: 81 fa ff 03 00 00 cmp $0x3ff,%edx 38c: 76 04 jbe 392 <__pthread_getspecific+0x12> 38e: 5d pop %ebp 38f: 31 c0 xor %eax,%eax 391: c3 ret 392: 89 d0 mov %edx,%eax 394: c1 e8 05 shr $0x5,%eax 397: 8d 0c 85 1c 01 00 00 lea 0x11c(,%eax,4),%ecx 39e: 65 8b 01 mov %gs:(%ecx),%eax 3a1: 85 c0 test %eax,%eax 3a3: 74 e9 je 38e <__pthread_getspecific+0xe> 3a5: 8b 04 d5 00 00 00 00 mov 0x0(,%edx,8),%eax 3ac: 85 c0 test %eax,%eax 3ae: 74 de je 38e <__pthread_getspecific+0xe> 3b0: 65 8b 01 mov %gs:(%ecx),%eax 3b3: 83 e2 1f and $0x1f,%edx 3b6: 8b 04 90 mov (%eax,%edx,4),%eax 3b9: 5d pop %ebp 3ba: c3 ret > Entering an uncontended pthread mutex should not be expensive Linux/x86: 00001020 <__pthread_self>: 1020: 55 push %ebp 1021: 89 e5 mov %esp,%ebp 1023: 65 a1 50 00 00 00 mov %gs:0x50,%eax 1029: 5d pop %ebp 102a: c3 ret 102b: 90 nop 102c: 8d 74 26 00 lea 0x0(%esi),%esi pretty lame, five instructions were only two are needed. 000004f0 <__pthread_mutex_lock>: .. too much to read through..but I think no kernel call.. - Jay From: jay.krell at cornell.edu To: dragisha at m3w.org; mika at async.async.caltech.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] userthreads vs. pthreads performance? Date: Sun, 28 Mar 2010 20:46:01 +0000 O(1) scheduling is not a new idea. Just look at NT and probably Solaris and probably all the other non-free systems (AIX, Irix, HP-UX, Tru64, VMS, etc.) Getting thread locals should not require a kernel call. It doesn't on NT. We can optimize this somewhat on most systems with __thread. I had that in briefly. Entering an uncontended pthread mutex should not be expensive -- at least no kernel call, but granted a call and atomic op. Two calls because of the C layer. But user threads pay for a call too of course. Maybe I should profile some of this.. - Jay > From: dragisha at m3w.org > To: mika at async.async.caltech.edu > Date: Sun, 28 Mar 2010 21:14:57 +0200 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > I remember reading (long time ago) about how these (FUTEXes) are > efficient in LINUX... Can I have your test code to try? > > On Sun, 2010-03-28 at 12:11 -0700, Mika Nystrom wrote: > > Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... > > > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > > >Which platform? > > > > > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > > >> Yep, sounds right. > > >> > > >> I was profiling some other thread-using code that slowed down > > >> enormously > > >> because of pthreads and it turned out the program was spending ~95% > > >> of its time in accessing the thread locals via one of the pthread_ > > >> functions. > > >> (The overhead of entering the kernel.) > > >-- > > >Dragi?a Duri? > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Mon Mar 29 11:38:18 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Mon, 29 Mar 2010 11:38:18 +0200 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: ,<1269771357.3671.7.camel@faramir.m3w.org> ,<20100328162424.670F91A20B7@async.async.caltech.edu> ,<9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> ,<1269801143.3671.17.camel@faramir.m3w.org> ,<20100328185740.38AFA1A20B9@async.async.caltech.edu> ,<1269802924.3671.18.camel@faramir.m3w.org> ,<20100328191116.CD9C71A20BB@async.async.caltech.edu> ,<1269803697.3671.19.camel@faramir.m3w.org> Message-ID: <1269855498.2369.1.camel@faramir.m3w.org> I can be wrong, but what FUTEX operations on Linux can have with pthread_getspecific? Operations are made on FUTEX, we know which and where, no need to ask for pthread_getspecific. Are we still talking about poor lock_mutex performance? On Mon, 2010-03-29 at 08:24 +0000, Jay K wrote: > Once you do throw in -fPIC and -shared, I have found __thread to be > significantly slower on Solaris/sparc and Linux/powerpc32, slower or > a wash on Linux/amd64, and twice as fast as pthread_getspecific on > Linux/x86. > I doesn't appear supported at all on Darwin, though > pthread_getspecific are very fast there (albeit not inlined). > I didn't test *BSD. -- Dragi?a Duri? From jay.krell at cornell.edu Mon Mar 29 15:10:36 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 29 Mar 2010 13:10:36 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269855498.2369.1.camel@faramir.m3w.org> References: , , <1269771357.3671.7.camel@faramir.m3w.org>, , <20100328162424.670F91A20B7@async.async.caltech.edu>, , <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, , <1269801143.3671.17.camel@faramir.m3w.org>, , <20100328185740.38AFA1A20B9@async.async.caltech.edu>, , <1269802924.3671.18.camel@faramir.m3w.org>, , <20100328191116.CD9C71A20BB@async.async.caltech.edu>, , <1269803697.3671.19.camel@faramir.m3w.org>, , <1269855498.2369.1.camel@faramir.m3w.org> Message-ID: Original was more general pthread vs. userthread and I thought pthread_getspecific was also indicted. - Jay > From: dragisha at m3w.org > To: jay.krell at cornell.edu > Date: Mon, 29 Mar 2010 11:38:18 +0200 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > I can be wrong, but what FUTEX operations on Linux can have with > pthread_getspecific? > > Operations are made on FUTEX, we know which and where, no need to ask > for pthread_getspecific. > > Are we still talking about poor lock_mutex performance? > > > On Mon, 2010-03-29 at 08:24 +0000, Jay K wrote: > > Once you do throw in -fPIC and -shared, I have found __thread to be > > significantly slower on Solaris/sparc and Linux/powerpc32, slower or > > a wash on Linux/amd64, and twice as fast as pthread_getspecific on > > Linux/x86. > > I doesn't appear supported at all on Darwin, though > > pthread_getspecific are very fast there (albeit not inlined). > > I didn't test *BSD. > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 29 16:42:17 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 29 Mar 2010 10:42:17 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> Message-ID: Guys, You are looking in the wrong place. The whole good thing about having a language-defined thread model is that we get to implement the thread primitives in the language run-time, and we can take advantage of language-specific semantics. There is no requirement that a Modula-3 mutex even map to a pthread_mutex_t. Java monitors are implemented in modern JVMs so that lock/unlock in the common case doesn't require any calls or any atomic operations! We can do much the same for Modula-3. My group at Purdue has done a fair amount of work on this in the context of Java, and when I can find the time I was hoping to do the same for Modula-3. The only requirement for M3 was that we have proper support for atomic ops in the language upon which to build the synchronisation primitives. We are close to having this now. Let me sketch a design: The common case is that a mutex is only ever manipulated by one thread (i.e., never shared), in which case it suffices to "bias" the mutex to that thread. Locking/unlocking is simply a matter of checking to see that the bias belongs to the current thread. No need for an atomic operation. If the thread already has the bias then locking/unlocking is simply a matter of setting a bit in the mutex. If another thread comes along and needs to lock the mutex then it must first revoke the bias of the owning thread. This can be expensive (assuming it occurs infrequently) and in our case probably means stopping the thread having the bias, revoking the bias, then restarting it. Another case is when a mutex is locked/unlocked by multiple threads but there is never contention (i.e., no thread tries to acquire while another thread holds). In this case we never need a wait queue for the mutex so we can simply store the lock owner in the mutex and test using atomic ops. Spinning is often useful here to avoid needing to inflate if contention ever does arise: if the thread holding the lock gets out of the mutex quickly then the spinner can move in quickly. After some number of spins we generally need to inflate the lock to allocate a wait queue (to avoiding hogging the processor). Finally, the case where many threads are contending on the inflated lock (with wait queue). The only question now is when to deflate. Our current heuristic is to deflate when the last thread releases the lock and notices that there are no other threads waiting. This seems to work well in practice, but of course there are pathological cases. Note that in no case have I mentioned the need for a pthread_mutex (though pthread locks/conditions are used to manage threads that must block on a Java quit queue). We ought to be able to do much the same in Modula-3. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 29 Mar 2010, at 04:24, Jay K wrote: > > We can optimize this somewhat on most systems with __thread. I had that in briefly > > > I did a little bit of testing here. > > > __thread takes a few forms, depending on which of -fPIC and -shared you use. > > Once you do throw in -fPIC and -shared, I have found __thread to be significantly slower on Solaris/sparc and Linux/powerpc32, slower or a wash on Linux/amd64, and twice as fast as pthread_getspecific on Linux/x86. > I doesn't appear supported at all on Darwin, though pthread_getspecific are very fast there (albeit not inlined). > I didn't test *BSD. > My testing was not very scientific. > However -fPIC and/or -shared imply a function call to access __thread variables. > That's probably the big factor. Without -fPIC/-shared, there is no function call. > > > If you are going to access them multiple times in a function then there is a probably an optimization to be had -- caching their address. If the variables are larger than a pointer, probably then also an optimization to be had. > > > We could do that first thing for certain -- PushFrame could return the address and PopFrame would be much faster. > However another angle here is to eliminate PushFrame/PopFrame, by using libunwind. I think we should look into libunwind for the next release. The thread locals will (mostly) remain but accesses to them greatly decline. > > > We compile everything -fPIC and -shared. > Some systems (libtool) compile things once that way and once not, providing pairs of libraries, depending on intended use. > > > I should point out also that userthreads have been greatly deoptimized in the current tree (by me, Tony approved), because they used to inline PushFrame/PopFrame, but they don't any longer. > > > (Historically on NT, __declspec(thread) only worked in code in the .exe or statically loaded by the .exe -- that is, not in .dlls loaded with LoadLibrary. However that limitation was removed in Vista. I expect __declspec(thread) is much faster than TlsGetValue, but I assume we'll support pre-Vista for a while longer so not interesting..) > > > - Jay > > > From: jay.krell at cornell.edu > To: dragisha at m3w.org; mika at async.async.caltech.edu > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] userthreads vs. pthreads performance? > Date: Mon, 29 Mar 2010 03:15:41 +0000 > > > Getting thread locals should not require a kernel call > > Indeed, on Linux/x86 it does not, looks pretty ok: > > 00000380 <__pthread_getspecific>: > 380: 55 push %ebp > 381: 89 e5 mov %esp,%ebp > 383: 8b 55 08 mov 0x8(%ebp),%edx > 386: 81 fa ff 03 00 00 cmp $0x3ff,%edx > 38c: 76 04 jbe 392 <__pthread_getspecific+0x12> > 38e: 5d pop %ebp > 38f: 31 c0 xor %eax,%eax > 391: c3 ret > > 392: 89 d0 mov %edx,%eax > 394: c1 e8 05 shr $0x5,%eax > 397: 8d 0c 85 1c 01 00 00 lea 0x11c(,%eax,4),%ecx > 39e: 65 8b 01 mov %gs:(%ecx),%eax > 3a1: 85 c0 test %eax,%eax > 3a3: 74 e9 je 38e <__pthread_getspecific+0xe> > 3a5: 8b 04 d5 00 00 00 00 mov 0x0(,%edx,8),%eax > 3ac: 85 c0 test %eax,%eax > 3ae: 74 de je 38e <__pthread_getspecific+0xe> > 3b0: 65 8b 01 mov %gs:(%ecx),%eax > 3b3: 83 e2 1f and $0x1f,%edx > 3b6: 8b 04 90 mov (%eax,%edx,4),%eax > 3b9: 5d pop %ebp > 3ba: c3 ret > > > > Entering an uncontended pthread mutex should not be expensive > > Linux/x86: > > 00001020 <__pthread_self>: > 1020: 55 push %ebp > 1021: 89 e5 mov %esp,%ebp > 1023: 65 a1 50 00 00 00 mov %gs:0x50,%eax > 1029: 5d pop %ebp > 102a: c3 ret > 102b: 90 nop > 102c: 8d 74 26 00 lea 0x0(%esi),%esi > > > pretty lame, five instructions were only two are needed. > > > 000004f0 <__pthread_mutex_lock>: > > .. too much to read through..but I think no kernel call.. > > - Jay > > From: jay.krell at cornell.edu > To: dragisha at m3w.org; mika at async.async.caltech.edu > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] userthreads vs. pthreads performance? > Date: Sun, 28 Mar 2010 20:46:01 +0000 > > O(1) scheduling is not a new idea. Just look at NT and probably Solaris and probably all the other non-free systems (AIX, Irix, HP-UX, Tru64, VMS, etc.) > > Getting thread locals should not require a kernel call. It doesn't on NT. We can optimize this somewhat on most systems with __thread. I had that in briefly. > > Entering an uncontended pthread mutex should not be expensive -- at least no kernel call, but granted a call and atomic op. Two calls because of the C layer. > But user threads pay for a call too of course. > > Maybe I should profile some of this.. > > - Jay > > > From: dragisha at m3w.org > > To: mika at async.async.caltech.edu > > Date: Sun, 28 Mar 2010 21:14:57 +0200 > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > > > I remember reading (long time ago) about how these (FUTEXes) are > > efficient in LINUX... Can I have your test code to try? > > > > On Sun, 2010-03-28 at 12:11 -0700, Mika Nystrom wrote: > > > Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... > > > > > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > > > >Which platform? > > > > > > > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > > > >> Yep, sounds right. > > > >> > > > >> I was profiling some other thread-using code that slowed down > > > >> enormously > > > >> because of pthreads and it turned out the program was spending ~95% > > > >> of its time in accessing the thread locals via one of the pthread_ > > > >> functions. > > > >> (The overhead of entering the kernel.) > > > >-- > > > >Dragi?a Duri? > > -- > > Dragi?a Duri? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 29 16:43:21 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 29 Mar 2010 10:43:21 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269855498.2369.1.camel@faramir.m3w.org> References: , <1269771357.3671.7.camel@faramir.m3w.org> , <20100328162424.670F91A20B7@async.async.caltech.edu> , <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> , <1269801143.3671.17.camel@faramir.m3w.org> , <20100328185740.38AFA1A20B9@async.async.caltech.edu> , <1269802924.3671.18.camel@faramir.m3w.org> , <20100328191116.CD9C71A20BB@async.async.caltech.edu> , <1269803697.3671.19.camel@faramir.m3w.org> <1269855498.2369.1.camel@faramir.m3w.org> Message-ID: PS. Our Java work shows that futexes don't really win against the bias/thin/fat lock scheme I sketched previously. On 29 Mar 2010, at 05:38, Dragi?a Duri? wrote: > I can be wrong, but what FUTEX operations on Linux can have with > pthread_getspecific? > > Operations are made on FUTEX, we know which and where, no need to ask > for pthread_getspecific. > > Are we still talking about poor lock_mutex performance? > > > On Mon, 2010-03-29 at 08:24 +0000, Jay K wrote: >> Once you do throw in -fPIC and -shared, I have found __thread to be >> significantly slower on Solaris/sparc and Linux/powerpc32, slower or >> a wash on Linux/amd64, and twice as fast as pthread_getspecific on >> Linux/x86. >> I doesn't appear supported at all on Darwin, though >> pthread_getspecific are very fast there (albeit not inlined). >> I didn't test *BSD. > -- > Dragi?a Duri? From rodney_bates at lcwb.coop Mon Mar 29 17:01:04 2010 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 29 Mar 2010 10:01:04 -0500 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> Message-ID: <4BB0C0B0.3010108@lcwb.coop> Tony Hosking wrote: > Guys, > > You are looking in the wrong place. The whole good thing about having a > language-defined thread model is that we get to implement the thread > primitives in the language run-time, and we can take advantage of > language-specific semantics. There is no requirement that a Modula-3 > mutex even map to a pthread_mutex_t. Java monitors are implemented in > modern JVMs so that lock/unlock in the common case doesn't require any > calls or any atomic operations! We can do much the same for Modula-3. > My group at Purdue has done a fair amount of work on this in the > context of Java, and when I can find the time I was hoping to do the > same for Modula-3. The only requirement for M3 was that we have proper > support for atomic ops in the language upon which to build the > synchronisation primitives. We are close to having this now. Let me > sketch a design: > > The common case is that a mutex is only ever manipulated by one thread Do you mean _almost_ only ever? -----^ Otherwise, why would it have been coded with a mutex at all? > (i.e., never shared), in which case it suffices to "bias" the mutex to > that thread. Locking/unlocking is simply a matter of checking to see > that the bias belongs to the current thread. No need for an atomic > operation. If the thread already has the bias then locking/unlocking is > simply a matter of setting a bit in the mutex. If another thread comes > along and needs to lock the mutex then it must first revoke the bias of > the owning thread. This can be expensive (assuming it occurs > infrequently) and in our case probably means stopping the thread having > the bias, revoking the bias, then restarting it. > > Another case is when a mutex is locked/unlocked by multiple threads but > there is never contention (i.e., no thread tries to acquire while -----------^ and _almost_ never? > another thread holds). In this case we never need a wait queue for the > mutex so we can simply store the lock owner in the mutex and test using > atomic ops. Spinning is often useful here to avoid needing to inflate > if contention ever does arise: if the thread holding the lock gets out > of the mutex quickly then the spinner can move in quickly. After some > number of spins we generally need to inflate the lock to allocate a wait > queue (to avoiding hogging the processor). > > Finally, the case where many threads are contending on the inflated lock > (with wait queue). The only question now is when to deflate. Our > current heuristic is to deflate when the last thread releases the lock > and notices that there are no other threads waiting. This seems to work > well in practice, but of course there are pathological cases. So does the implementation dynamically detect which case holds and choose between the schemes? What are the criteria? > > Note that in no case have I mentioned the need for a pthread_mutex > (though pthread locks/conditions are used to manage threads that must > block on a Java quit queue). > > We ought to be able to do much the same in Modula-3. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > From mika at async.async.caltech.edu Mon Mar 29 17:20:25 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Mon, 29 Mar 2010 08:20:25 -0700 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <4BB0C0B0.3010108@lcwb.coop> References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> <4BB0C0B0.3010108@lcwb.coop> Message-ID: <20100329152025.630591A20C2@async.async.caltech.edu> "Rodney M. Bates" writes: > ... >> >> The common case is that a mutex is only ever manipulated by one thread >Do you mean _almost_ only ever? -----^ Otherwise, why would it have been >coded with a mutex at all? ... In a single-threaded program.... lots of mutexes hiding in various software libraries (e.g., Rd/Wr)... Mika From hosking at cs.purdue.edu Mon Mar 29 19:26:28 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 29 Mar 2010 13:26:28 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <4BB0C0B0.3010108@lcwb.coop> References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> <4BB0C0B0.3010108@lcwb.coop> Message-ID: <3716771F-D63C-4ABC-8977-3B7F509BEB88@cs.purdue.edu> On 29 Mar 2010, at 11:01, Rodney M. Bates wrote: > > > Tony Hosking wrote: >> Guys, >> You are looking in the wrong place. The whole good thing about having a language-defined thread model is that we get to implement the thread primitives in the language run-time, and we can take advantage of language-specific semantics. There is no requirement that a Modula-3 mutex even map to a pthread_mutex_t. Java monitors are implemented in modern JVMs so that lock/unlock in the common case doesn't require any calls or any atomic operations! We can do much the same for Modula-3. My group at Purdue has done a fair amount of work on this in the context of Java, and when I can find the time I was hoping to do the same for Modula-3. The only requirement for M3 was that we have proper support for atomic ops in the language upon which to build the synchronisation primitives. We are close to having this now. Let me sketch a design: >> The common case is that a mutex is only ever manipulated by one thread > Do you mean _almost_ only ever? -----^ Otherwise, why would it have been > coded with a mutex at all? You might have a thread-safe library that is used by a single-thread application. >> (i.e., never shared), in which case it suffices to "bias" the mutex to that thread. Locking/unlocking is simply a matter of checking to see that the bias belongs to the current thread. No need for an atomic operation. If the thread already has the bias then locking/unlocking is simply a matter of setting a bit in the mutex. If another thread comes along and needs to lock the mutex then it must first revoke the bias of the owning thread. This can be expensive (assuming it occurs infrequently) and in our case probably means stopping the thread having the bias, revoking the bias, then restarting it. >> Another case is when a mutex is locked/unlocked by multiple threads but there is never contention (i.e., no thread tries to acquire while > -----------^ and _almost_ never? > >> another thread holds). In this case we never need a wait queue for the mutex so we can simply store the lock owner in the mutex and test using atomic ops. Spinning is often useful here to avoid needing to inflate if contention ever does arise: if the thread holding the lock gets out of the mutex quickly then the spinner can move in quickly. After some number of spins we generally need to inflate the lock to allocate a wait queue (to avoiding hogging the processor). >> Finally, the case where many threads are contending on the inflated lock (with wait queue). The only question now is when to deflate. Our current heuristic is to deflate when the last thread releases the lock and notices that there are no other threads waiting. This seems to work well in practice, but of course there are pathological cases. > > So does the implementation dynamically detect which case holds and > choose between the schemes? What are the criteria? Short answer, yes. Criteria much as I described. The only trick thing is choosing when to deflate. For more on this topic take a look also at: http://blogs.azulsystems.com/cliff/. Azul independently developed similar strategies. > >> Note that in no case have I mentioned the need for a pthread_mutex (though pthread locks/conditions are used to manage threads that must block on a Java quit queue). >> We ought to be able to do much the same in Modula-3. >> Antony Hosking | Associate Professor | Computer Science | Purdue University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> From dragisha at m3w.org Tue Mar 30 01:47:36 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Tue, 30 Mar 2010 01:47:36 +0200 Subject: [M3devel] user vs. kernel Message-ID: <1269906456.3029.4.camel@faramir.m3w.org> I am following this thread, and I have just few points to highlight: 1. battle user vs. kernel threads was finished long time ago. let's not pull back Modula-3 back where it was before last few years of development by doing things backward; 2. bias et al is nice thing, and I believe we will have it in no time; as compared to total development time of cm3. -- Dragi?a Duri? From jay.krell at cornell.edu Tue Mar 30 08:37:40 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 30 Mar 2010 06:37:40 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <3716771F-D63C-4ABC-8977-3B7F509BEB88@cs.purdue.edu> References: , , <1269771357.3671.7.camel@faramir.m3w.org>, , <20100328162424.670F91A20B7@async.async.caltech.edu>, , <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, , <1269801143.3671.17.camel@faramir.m3w.org>, , <20100328185740.38AFA1A20B9@async.async.caltech.edu>, , <1269802924.3671.18.camel@faramir.m3w.org>, , <20100328191116.CD9C71A20BB@async.async.caltech.edu>, , <1269803697.3671.19.camel@faramir.m3w.org> , , <4BB0C0B0.3010108@lcwb.coop>, <3716771F-D63C-4ABC-8977-3B7F509BEB88@cs.purdue.edu> Message-ID: > The only requirement for M3 was that we have proper support for > atomic ops in the language upon which to build the synchronisation > primitives. We are close to having this now. Or surely implement them in C.. Anyway, I believe m3back now (a few weeks) has parity with or exceeds the gcc backend. Depending on interpretation. The gcc backend only accepts some memory orders. m3back accepts them all and interprets them all as sequential. I have more testing and optimization to do, but I think it is all there and probably works. Optimization in particular is that add/sub/xor don't need to use compare/exchange/retry loops. And load/store could certainly be more efficient such as just using one exchange to do a store, instead of a fence on either side. I think it is also probably reasonable to defined variants that don't return the new value. That way or and and can be provided more efficiently -- without a compare/exchange/retry loop. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 30 14:51:34 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 30 Mar 2010 08:51:34 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: , , <1269771357.3671.7.camel@faramir.m3w.org>, , <20100328162424.670F91A20B7@async.async.caltech.edu>, , <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, , <1269801143.3671.17.camel@faramir.m3w.org>, , <20100328185740.38AFA1A20B9@async.async.caltech.edu>, , <1269802924.3671.18.camel@faramir.m3w.org>, , <20100328191116.CD9C71A20BB@async.async.caltech.edu>, , <1269803697.3671.19.camel@faramir.m3w.org> , , <4BB0C0B0.3010108@lcwb.coop>, <3716771F-D63C-4ABC-8977-3B7F509BEB88@cs.purdue.edu> Message-ID: On 30 Mar 2010, at 02:37, Jay K wrote: > > The only requirement for M3 was that we have proper support for > > atomic ops in the language upon which to build the synchronisation > > primitives. We are close to having this now. > > > Or surely implement them in C.. We want them inline. > Anyway, I believe m3back now (a few weeks) has parity with or exceeds the gcc backend. > Depending on interpretation. > > > The gcc backend only accepts some memory orders. > > > m3back accepts them all and interprets them all as sequential. same as gcc. > > > I have more testing and optimization to do, but I think it is all there > and probably works. Optimization in particular is that add/sub/xor > don't need to use compare/exchange/retry loops. > > > And load/store could certainly be more efficient such as just > using one exchange to do a store, instead of a fence on either side. > > > I think it is also probably reasonable to defined variants that > don't return the new value. That way or and and can be provided > more efficiently -- without a compare/exchange/retry loop. > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 31 02:53:27 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 00:53:27 +0000 Subject: [M3devel] m3core/src/types/Unicode.m3 var to const Message-ID: This is right, right? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 1.txt URL: From hosking at cs.purdue.edu Wed Mar 31 04:39:25 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 30 Mar 2010 22:39:25 -0400 Subject: [M3devel] m3core/src/types/Unicode.m3 var to const In-Reply-To: References: Message-ID: <91EA5B0C-96CA-4C36-B9C1-DEF2BC5D29CB@cs.purdue.edu> Yes, I think that looks OK. The original is very weird M3 code! Also, can you now make it a safe module instead of UNSAFE? (No use of ADR anymore). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 30 Mar 2010, at 20:53, Jay K wrote: > This is right, right? > > - Jay > > > > > > <1.txt> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 31 05:19:57 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 03:19:57 +0000 Subject: [M3devel] m3core/src/types/Unicode.m3 var to const In-Reply-To: <91EA5B0C-96CA-4C36-B9C1-DEF2BC5D29CB@cs.purdue.edu> References: , <91EA5B0C-96CA-4C36-B9C1-DEF2BC5D29CB@cs.purdue.edu> Message-ID: "I think I can". Just use arrays and indices more and pointers to elements less/not at all. Granted, moving the division is a big speed deoptimization since it is no longer by a constant (unless compiler is quite smart and does "partial inlining", and NT386 compiler is not, very few compilers are smart enough to see this). It might be worth a) restoring that small part b) having two functions, one for 2-tuples, one for 3-tuples, or even c) surely we con't need to handcode binary search here in the first place there is some generic reusable version? But that is tangential to var vs. const. I was also considering adding: Int8, UInt8, Int16, UInt16, UInt32, UInt64. In particular, in m3back, I think I'll be using UInt16 in tables -- CodeView 4.0 format uses UINT16 to represent types (CodeView 5.0 uses UINT32). In reality due to padding/alignment, UINT8/16 probably have no space savings, but they will get range checks..but I'll also have to add a range check when I "allocate" a new type. "That" is "why" I was looking in this directory. - Jay From: hosking at cs.purdue.edu Date: Tue, 30 Mar 2010 22:39:25 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] m3core/src/types/Unicode.m3 var to const Yes, I think that looks OK. The original is very weird M3 code! Also, can you now make it a safe module instead of UNSAFE? (No use of ADR anymore). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 30 Mar 2010, at 20:53, Jay K wrote: This is right, right? - Jay <1.txt> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 31 06:41:14 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 04:41:14 +0000 Subject: [M3devel] m3back directions? Message-ID: I'm curious what, if anything, people are interested in in m3back? There are several mostly independent directions: - remove it; use the gcc backend or other (burg, llvm, generate C) - expand to support other targets, AMD64_*, including AMD64_NT m3objfile would need macho/elf support for non-NT - expand to generate good debugging information for Microsoft debuggers - various smaller/larger optimizations inlining? Inlining seems like the most lacking optimization and thinking about it, it doesn't actually seem that hard to do, at least a little bit. I'm thinking for now of working on the debugging information. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 31 10:02:48 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 08:02:48 +0000 Subject: [M3devel] generic binary search in standard Modula-3 library? In-Reply-To: References: , , <91EA5B0C-96CA-4C36-B9C1-DEF2BC5D29CB@cs.purdue.edu>, Message-ID: > c) surely we con't need to handcode binary search here in the first place there is some generic reusable version? The closest precedent I can find is: http://modula3.elegosoft.com/cm3/doc/help/gen_html/libm3/src/sort/ArraySort.ig.html so I propose *something like*: m3core/src/binarysearch/BinarySearch.ig. PROCEDURE BinarySearch(READONLY key: Elem.T; READONLY a: ARRAY OF Elem.T; cmp := Elem.Compare): INTEGER; along with: PROCEDURE LowerBound(READONLY key: Elem.T; READONLY a: ARRAY OF Elem.T; cmp := Elem.Compare): INTEGER; PROCEDURE UpperBound(READONLY key: Elem.T; READONLY a: ARRAY OF Elem.T; cmp := Elem.Compare): INTEGER; PROCEDURE EqualRange(READONLY key: Elem.T; READONLY a: ARRAY OF Elem.T; VAR lower, upper: INTEGER; cmp := Elem.Compare); using -1 for not found? LowerBound, UpperBound, EqualRange are closely related to BinarySearch. When the array contains adjacent equal/equivalent elements, they let you find the first, last or first and last, whereas "regular" BinarySearch finds an arbitrary one. See the C++ STL. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 31 Mar 2010 03:19:57 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] m3core/src/types/Unicode.m3 var to const "I think I can". Just use arrays and indices more and pointers to elements less/not at all. Granted, moving the division is a big speed deoptimization since it is no longer by a constant (unless compiler is quite smart and does "partial inlining", and NT386 compiler is not, very few compilers are smart enough to see this). It might be worth a) restoring that small part b) having two functions, one for 2-tuples, one for 3-tuples, or even c) surely we con't need to handcode binary search here in the first place there is some generic reusable version? But that is tangential to var vs. const. I was also considering adding: Int8, UInt8, Int16, UInt16, UInt32, UInt64. In particular, in m3back, I think I'll be using UInt16 in tables -- CodeView 4.0 format uses UINT16 to represent types (CodeView 5.0 uses UINT32). In reality due to padding/alignment, UINT8/16 probably have no space savings, but they will get range checks..but I'll also have to add a range check when I "allocate" a new type. "That" is "why" I was looking in this directory. - Jay From: hosking at cs.purdue.edu Date: Tue, 30 Mar 2010 22:39:25 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] m3core/src/types/Unicode.m3 var to const Yes, I think that looks OK. The original is very weird M3 code! Also, can you now make it a safe module instead of UNSAFE? (No use of ADR anymore). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 30 Mar 2010, at 20:53, Jay K wrote: This is right, right? - Jay <1.txt> -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Wed Mar 31 11:37:31 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Wed, 31 Mar 2010 11:37:31 +0200 Subject: [M3devel] m3back directions? In-Reply-To: References: Message-ID: <1270028251.3114.4.camel@faramir.m3w.org> It is started job and useable right now. IMO - it's prolonged and thriving existence shows directions and maintains secure interface for multiple backends - LLVM for example. Thus said, my opinion is: a) keep it, and make it excellent working x86 backend alternative. b) AMD64, as natural successor to x86 - maybe, if it's not too much work. Even without it - m3back can probably work for many x86 appliances present - for years to come. c) debug - of course it's good idea. But, if it's alternative, one can debug her code using gcc, then switch to m3back build for productivity. d) nothing too big of optimizations, although inlining would be sweet. dd On Wed, 2010-03-31 at 04:41 +0000, Jay K wrote: > I'm curious what, if anything, people are interested in in m3back? > There are several mostly independent directions: > - remove it; use the gcc backend or other (burg, llvm, generate C) > - expand to support other targets, AMD64_*, including AMD64_NT > m3objfile would need macho/elf support for non-NT > - expand to generate good debugging information for Microsoft > debuggers > - various smaller/larger optimizations > inlining? Inlining seems like the most lacking optimization and > thinking about it, it doesn't actually seem that hard to do, at least > a little bit. -- Dragi?a Duri? From wagner at elegosoft.com Wed Mar 31 12:18:30 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Wed, 31 Mar 2010 12:18:30 +0200 Subject: [M3devel] m3back directions? In-Reply-To: References: Message-ID: <20100331121830.22q33okrokkwsgwk@mail.elegosoft.com> Quoting Jay K : > I'm curious what, if anything, people are interested in in m3back? > > There are several mostly independent directions: > > - remove it; use the gcc backend or other (burg, llvm, generate C) > > - expand to support other targets, AMD64_*, including AMD64_NT > > m3objfile would need macho/elf support for non-NT It would be great if we could use the integrated backend for other targets, too. Adding ELF support will be a lot of work, but it's probably worth it. Olaf > - expand to generate good debugging information for Microsoft debuggers > > - various smaller/larger optimizations > > inlining? Inlining seems like the most lacking optimization and > thinking about it, it doesn't actually seem that hard to do, at > least a little bit. > > > > > > I'm thinking for now of working on the debugging information. > > > > > > - Jay > > > From jay.krell at cornell.edu Wed Mar 31 13:01:17 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 11:01:17 +0000 Subject: [M3devel] m3back directions? In-Reply-To: <20100331121830.22q33okrokkwsgwk@mail.elegosoft.com> References: , <20100331121830.22q33okrokkwsgwk@mail.elegosoft.com> Message-ID: > remove it? Oops, did I say that? :) Well, we should definitely do some of the others if are going to keep it. :) I'm leaning toward Microsoft CodeView debugging information currently. It should be the easiest and provide a lot of value. Though it is actually in m3objfile. I forgot to mention: It is already I think very portable to all 32bit x86 platforms. The real work work would be in m3objfile. A little in m3back regarding calling conventions perhaps. Maybe a little around munging function names for "-shared -fPIC" too. Like appending "@plt" in function calls or somesuch. - Jay > Date: Wed, 31 Mar 2010 12:18:30 +0200 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] m3back directions? > > Quoting Jay K : > > > I'm curious what, if anything, people are interested in in m3back? > > > > There are several mostly independent directions: > > > > - remove it; use the gcc backend or other (burg, llvm, generate C) > > > > - expand to support other targets, AMD64_*, including AMD64_NT > > > > m3objfile would need macho/elf support for non-NT > > It would be great if we could use the integrated backend for other > targets, too. Adding ELF support will be a lot of work, but it's probably > worth it. > > Olaf > > > - expand to generate good debugging information for Microsoft debuggers > > > > - various smaller/larger optimizations > > > > inlining? Inlining seems like the most lacking optimization and > > thinking about it, it doesn't actually seem that hard to do, at > > least a little bit. > > > > > > > > > > > > I'm thinking for now of working on the debugging information. > > > > > > > > > > > > - Jay > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 31 13:15:08 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 11:15:08 +0000 Subject: [M3devel] changing m3middle/m3back to have multiple passes? Message-ID: I'm thinking m3back could use multiple passes. There a few reasons, things that would be easier/possible if it had them. I know adding passes will slow it down, but I suspect it will still be about as pleasantly fast as it is now. The optimizations I have in mind: inlining, including where function definitions come after the call I think inlining where the "small" functions come first is doable in pass, but not if uses come before calls. not saving unused registers This only a small optimization and could probably be done in one pass if m3objfile got a "memmove" operation. Well, I already have the optimization in somewhat, but it involves nop-ing out code instead of removing it completely. dead store removal, and then removing the resulting dead code to compute the value possibly better register allocation -- e.g. based on noticing which variables are used often, or based on how they are used, for example if a variable (or expression/temporary) ends serving a shift count, we should favor putting it in ecx up front, instead of moving it to ecx later, but we don't generally know this till too late As I understand, m3middle already has the notion of "recording" and "playing back" calls to the M3CG interface. That seems like a good starting point? Some optimizations can be made via "CG to CG" transforms. However I think in general I think the required design would include: - ability to add in "private operations"; maybe - ability to add "private data" to existing operations A very simple one is annotate functions with required frame size. This isn't currently known until the end of the function. - clearly, ability to remove operations Can/should we somehow augment m3middle in support of this line of thinking? Or is that just not the way to go? Each backend should have its own internal representation and loop over it? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 31 16:02:36 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 31 Mar 2010 10:02:36 -0400 Subject: [M3devel] m3back directions? In-Reply-To: References: Message-ID: <376BB9E6-AC9B-4110-AD22-83F8186BE5BF@cs.purdue.edu> On 31 Mar 2010, at 00:41, Jay K wrote: > I'm curious what, if anything, people are interested in in m3back? Simple. Fast. It is target-dependent so optimising there is misguided effort unless the opts are target-dependent (instruction selection, register allocation!). We don't have resources to maintain a sophisticated backend. Better to rely on other tools. There was a Linux version in pm3 that we should be able to model on. > There are several mostly independent directions: > - remove it; use the gcc backend or other (burg, llvm, generate C) > - expand to support other targets, AMD64_*, including AMD64_NT > m3objfile would need macho/elf support for non-NT > - expand to generate good debugging information for Microsoft debuggers > - various smaller/larger optimizations > inlining? Inlining seems like the most lacking optimization and thinking about it, it doesn't actually seem that hard to do, at least a little bit. > > > I'm thinking for now of working on the debugging information. > > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 31 18:08:38 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 31 Mar 2010 12:08:38 -0400 Subject: [M3devel] m3back directions? In-Reply-To: <20100331121830.22q33okrokkwsgwk@mail.elegosoft.com> References: <20100331121830.22q33okrokkwsgwk@mail.elegosoft.com> Message-ID: On 31 Mar 2010, at 06:18, wagner at elegosoft.com wrote: > Quoting Jay K : > >> I'm curious what, if anything, people are interested in in m3back? >> >> There are several mostly independent directions: >> >> - remove it; use the gcc backend or other (burg, llvm, generate C) >> >> - expand to support other targets, AMD64_*, including AMD64_NT >> >> m3objfile would need macho/elf support for non-NT > > It would be great if we could use the integrated backend for other > targets, too. Adding ELF support will be a lot of work, but it's probably > worth it. Have you looked at the pm3 Linux support? > > Olaf > >> - expand to generate good debugging information for Microsoft debuggers >> >> - various smaller/larger optimizations >> >> inlining? Inlining seems like the most lacking optimization and thinking about it, it doesn't actually seem that hard to do, at least a little bit. >> >> >> >> >> >> I'm thinking for now of working on the debugging information. >> >> >> >> >> >> - Jay >> >> >> > > From wagner at elegosoft.com Mon Mar 1 11:21:52 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 01 Mar 2010 11:21:52 +0100 Subject: [M3devel] Tinderbox AMD64_LINUX and errors in libm3 tests Message-ID: <20100301112152.v325as9n4c48coo0@mail.elegosoft.com> Hi, after some script merges from the release branch to the trunk and adaptation of the tinderbox scripts on birch this platform finally seems to build and test OK again. However, I noticed a compilation error in the libm3 tests: --- tests in /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX--- new source -> compiling PathnameTests.i3 new source -> compiling PathnameTests.m3 new source -> compiling Subr.i3 new source -> compiling TextSubrTbl.i3 new source -> compiling TextSubrTbl.m3 new source -> compiling OSTest.m3 "../src/OSTest.m3", line 298: incompatible types (i) 1 error encountered compilation failed => not building program "OSTest" performing pathname-tests... /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX Fatal Error: package build failed cm3 returned 2 Would anybody care to fix that? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Mon Mar 1 11:42:57 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 1 Mar 2010 10:42:57 +0000 Subject: [M3devel] Tinderbox AMD64_LINUX and errors in libm3 tests In-Reply-To: <20100301112152.v325as9n4c48coo0@mail.elegosoft.com> References: <20100301112152.v325as9n4c48coo0@mail.elegosoft.com> Message-ID: I only ever use m3-sys/m3ests. I haven't learned to use any of the other tests. Surely this is longint. I'll look/fix shortly, though we could really use more participation. - Jay > Date: Mon, 1 Mar 2010 11:21:52 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: [M3devel] Tinderbox AMD64_LINUX and errors in libm3 tests > > Hi, > > after some script merges from the release branch to the trunk and > adaptation of the tinderbox scripts on birch this platform finally > seems to build and test OK again. > > However, I noticed a compilation error in the libm3 tests: > > --- tests in > /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX--- > new source -> compiling PathnameTests.i3 > new source -> compiling PathnameTests.m3 > new source -> compiling Subr.i3 > new source -> compiling TextSubrTbl.i3 > new source -> compiling TextSubrTbl.m3 > new source -> compiling OSTest.m3 > "../src/OSTest.m3", line 298: incompatible types (i) > 1 error encountered > compilation failed => not building program "OSTest" > performing pathname-tests... > /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX > Fatal Error: package build failed > > cm3 returned 2 > > Would anybody care to fix that? > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Mon Mar 1 11:59:27 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 01 Mar 2010 11:59:27 +0100 Subject: [M3devel] Tinderbox AMD64_LINUX and errors in libm3 tests In-Reply-To: References: <20100301112152.v325as9n4c48coo0@mail.elegosoft.com> Message-ID: <20100301115927.xn0vc5tnggksgggc@mail.elegosoft.com> Quoting Jay K : > I only ever use m3-sys/m3ests. I haven't learned to use any of the > other tests. These tests are included in the package tests. Each package may have one or more associated test directories. The tests are run after the package has been built in the test-all-pkgs jobs, both in tinderbox http://www.modula3.com/cm3/logs/package-status.html http://www.modula3.com/cm3/logs/cm3-pkg-report-AMD64_LINUX-2010-03-01-03-31-05.html http://www.modula3.com/cm3/logs/cm3-pkg-report-FreeBSD4-2010-03-01-04-53-25.html ... and in Hudson, for example http://hudson.modula3.com:8080/job/cm3-test-all-pkgs-AMD64_LINUX/ http://hudson.modula3.com:8080/job/cm3-test-all-pkgs-PPC_DARWIN/ http://hudson.modula3.com:8080/job/cm3-test-all-pkgs-FreeBSD4/ ... > Surely this is longint. I'll look/fix shortly, though we could > really use more participation. Great. Thanks, Olaf > - Jay > >> Date: Mon, 1 Mar 2010 11:21:52 +0100 >> From: wagner at elegosoft.com >> To: m3devel at elegosoft.com >> Subject: [M3devel] Tinderbox AMD64_LINUX and errors in libm3 tests >> >> Hi, >> >> after some script merges from the release branch to the trunk and >> adaptation of the tinderbox scripts on birch this platform finally >> seems to build and test OK again. >> >> However, I noticed a compilation error in the libm3 tests: >> >> --- tests in >> /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX--- >> new source -> compiling PathnameTests.i3 >> new source -> compiling PathnameTests.m3 >> new source -> compiling Subr.i3 >> new source -> compiling TextSubrTbl.i3 >> new source -> compiling TextSubrTbl.m3 >> new source -> compiling OSTest.m3 >> "../src/OSTest.m3", line 298: incompatible types (i) >> 1 error encountered >> compilation failed => not building program "OSTest" >> performing pathname-tests... >> /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX >> Fatal Error: package build failed >> >> cm3 returned 2 >> >> Would anybody care to fix that? >> >> Olaf >> -- >> Olaf Wagner -- elego Software Solutions GmbH >> Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany >> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 >> http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin >> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 >> > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 2 14:16:03 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 2 Mar 2010 13:16:03 +0000 Subject: [M3devel] FW: optimizing range checks? In-Reply-To: <20100302125931.619312474001@birch.elegosoft.com> References: <20100302125931.619312474001@birch.elegosoft.com> Message-ID: Hey that's pretty clever. It costs a register, but given: if (b >= constantX|| b <= -constantY) a = 0; The C compiler instead does something like: if ((b + constantY - 1) > (constantX + constantY - 1)) a = 0; This is something the front end could do in many cases.? Adding a constant to eliminate one of the compares and branches is a win. If an x86 compiler will give up a register for this, then it is probably a win everywhere. Granted, it probably requires silent overflow. Oh well. - Jay > Date: Tue, 2 Mar 2010 13:59:31 +0000 > To: m3commit at elegosoft.com > From: jkrell at elego.de > Subject: [M3commit] CVS Update: cm3 > > CVSROOT: /usr/cvs > Changes by: jkrell at birch. 10/03/02 13:59:31 > > Modified files: > cm3/m3-libs/m3core/src/Csupport/Common/: hand.c > > Log message: > remove m3_shift64, which for the record was: > uint64 __stdcall m3_shift64(uint64 a, int b) > { > if (b >= 64 || b <= -64) > a = 0; > else if (b > 0) > a <<= b; > else if (b < 0) > a >>= -b; > return a; > } > > _m3_shift64 at 12: > 00000000: 55 push ebp > 00000001: 8B EC mov ebp,esp > 00000003: 8B 4D 10 mov ecx,dword ptr [ebp+10h] > 00000006: 8D 41 3F lea eax,[ecx+3Fh] > 00000009: 83 F8 7E cmp eax,7Eh > 0000000C: 77 1C ja 0000002A > 0000000E: 8B 45 08 mov eax,dword ptr [ebp+8] > 00000011: 8B 55 0C mov edx,dword ptr [ebp+0Ch] > 00000014: 85 C9 test ecx,ecx > 00000016: 7E 07 jle 0000001F > 00000018: E8 00 00 00 00 call __allshl > 0000001D: EB 0F jmp 0000002E > 0000001F: 7D 0D jge 0000002E > 00000021: F7 D9 neg ecx > 00000023: E8 00 00 00 00 call __aullshr > 00000028: EB 04 jmp 0000002E > 0000002A: 33 C0 xor eax,eax > 0000002C: 33 D2 xor edx,edx > 0000002E: 5D pop ebp > 0000002F: C2 0C 00 ret 0Ch > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 2 15:04:47 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 2 Mar 2010 14:04:47 +0000 Subject: [M3devel] overshifting? Message-ID: PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); I'm guessing the warning is all you're supposed to get here? C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 --- building in NT386 --- new source -> compiling Main.m3 "..\Main.m3", line 894: warning: value out of range *** *** runtime error: *** An enumeration or subrange value was out of range. *** file "..\src\TWordN.m3", line 129 *** Stack trace: FP PC Procedure --------- --------- ------------------------------- 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti nWord\Shift.mg 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 ......... ......... ... more frames ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 19:54:54 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 13:54:54 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: Message-ID: <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. On 2 Mar 2010, at 09:04, Jay K wrote: > PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > > > I'm guessing the warning is all you're supposed to get here? > > > C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > --- building in NT386 --- > new source -> compiling Main.m3 > "..\Main.m3", line 894: warning: value out of range > > *** > *** runtime error: > *** An enumeration or subrange value was out of range. > *** file "..\src\TWordN.m3", line 129 > *** > Stack trace: > FP PC Procedure > --------- --------- ------------------------------- > 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > nWord\Shift.mg > 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > ......... ......... ... more frames ... > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 19:55:38 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 13:55:38 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: Message-ID: <06913ED3-F44E-4A00-822C-8DC1F06E6C06@cs.purdue.edu> PS The warning is there to say that there might be a runtime error. On 2 Mar 2010, at 09:04, Jay K wrote: > PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > > > I'm guessing the warning is all you're supposed to get here? > > > C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > --- building in NT386 --- > new source -> compiling Main.m3 > "..\Main.m3", line 894: warning: value out of range > > *** > *** runtime error: > *** An enumeration or subrange value was out of range. > *** file "..\src\TWordN.m3", line 129 > *** > Stack trace: > FP PC Procedure > --------- --------- ------------------------------- > 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > nWord\Shift.mg > 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > ......... ......... ... more frames ... > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 19:58:05 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 13:58:05 -0500 Subject: [M3devel] FW: optimizing range checks? In-Reply-To: References: <20100302125931.619312474001@birch.elegosoft.com> Message-ID: The front-end is supposed to be machine-independent. It has no idea what the underlying machine will do on overflow. On 2 Mar 2010, at 08:16, Jay K wrote: > Hey that's pretty clever. > It costs a register, but given: > > if (b >= constantX|| b <= -constantY) > a = 0; > > The C compiler instead does something like: > if ((b + constantY - 1) > (constantX + constantY - 1)) > a = 0; > > This is something the front end could do in many cases.? > Adding a constant to eliminate one of the compares and branches is a win. > If an x86 compiler will give up a register for this, then it is probably a win everywhere. > > Granted, it probably requires silent overflow. Oh well. > > - Jay > > > Date: Tue, 2 Mar 2010 13:59:31 +0000 > > To: m3commit at elegosoft.com > > From: jkrell at elego.de > > Subject: [M3commit] CVS Update: cm3 > > > > CVSROOT: /usr/cvs > > Changes by: jkrell at birch. 10/03/02 13:59:31 > > > > Modified files: > > cm3/m3-libs/m3core/src/Csupport/Common/: hand.c > > > > Log message: > > remove m3_shift64, which for the record was: > > uint64 __stdcall m3_shift64(uint64 a, int b) > > { > > if (b >= 64 || b <= -64) > > a = 0; > > else if (b > 0) > > a <<= b; > > else if (b < 0) > > a >>= -b; > > return a; > > } > > > > _m3_shift64 at 12: > > 00000000: 55 push ebp > > 00000001: 8B EC mov ebp,esp > > 00000003: 8B 4D 10 mov ecx,dword ptr [ebp+10h] > > 00000006: 8D 41 3F lea eax,[ecx+3Fh] > > 00000009: 83 F8 7E cmp eax,7Eh > > 0000000C: 77 1C ja 0000002A > > 0000000E: 8B 45 08 mov eax,dword ptr [ebp+8] > > 00000011: 8B 55 0C mov edx,dword ptr [ebp+0Ch] > > 00000014: 85 C9 test ecx,ecx > > 00000016: 7E 07 jle 0000001F > > 00000018: E8 00 00 00 00 call __allshl > > 0000001D: EB 0F jmp 0000002E > > 0000001F: 7D 0D jge 0000002E > > 00000021: F7 D9 neg ecx > > 00000023: E8 00 00 00 00 call __aullshr > > 00000028: EB 04 jmp 0000002E > > 0000002A: 33 C0 xor eax,eax > > 0000002C: 33 D2 xor edx,edx > > 0000002E: 5D pop ebp > > 0000002F: C2 0C 00 ret 0Ch > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 20:50:27 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 14:50:27 -0500 Subject: [M3devel] overshifting? In-Reply-To: <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu> References: <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu> <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu> <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu> Message-ID: <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu> Oh, yes, of course your backend should not blow up on this. On 2 Mar 2010, at 14:46, Tony Hosking wrote: > P.S. Here are the signatures of Shift, RightShift, and LeftShift. > > PROCEDURE Shift (x: T; n: INTEGER): T; > For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n > 0 is like multiplying by 2^(n) > PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > = Shift (x, n) > PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > = Shift (x, -n) > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 2 Mar 2010, at 14:44, Tony Hosking wrote: > >> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. >> >> So, in fact the run-time error is correct! 630 is not in the range [0..63]. >> >> On 2 Mar 2010, at 13:54, Tony Hosking wrote: >> >>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >>> >>> On 2 Mar 2010, at 09:04, Jay K wrote: >>> >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >>>> >>>> >>>> I'm guessing the warning is all you're supposed to get here? >>>> >>>> >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >>>> --- building in NT386 --- >>>> new source -> compiling Main.m3 >>>> "..\Main.m3", line 894: warning: value out of range >>>> >>>> *** >>>> *** runtime error: >>>> *** An enumeration or subrange value was out of range. >>>> *** file "..\src\TWordN.m3", line 129 >>>> *** >>>> Stack trace: >>>> FP PC Procedure >>>> --------- --------- ------------------------------- >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >>>> nWord\Shift.mg >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >>>> ......... ......... ... more frames ... >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 20:44:21 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 14:44:21 -0500 Subject: [M3devel] overshifting? In-Reply-To: <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu> References: <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu> Message-ID: <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. So, in fact the run-time error is correct! 630 is not in the range [0..63]. On 2 Mar 2010, at 13:54, Tony Hosking wrote: > I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > > On 2 Mar 2010, at 09:04, Jay K wrote: > >> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >> >> >> I'm guessing the warning is all you're supposed to get here? >> >> >> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >> --- building in NT386 --- >> new source -> compiling Main.m3 >> "..\Main.m3", line 894: warning: value out of range >> >> *** >> *** runtime error: >> *** An enumeration or subrange value was out of range. >> *** file "..\src\TWordN.m3", line 129 >> *** >> Stack trace: >> FP PC Procedure >> --------- --------- ------------------------------- >> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >> nWord\Shift.mg >> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >> ......... ......... ... more frames ... >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 20:46:36 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 14:46:36 -0500 Subject: [M3devel] overshifting? In-Reply-To: <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu> References: <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu> <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu> Message-ID: <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu> P.S. Here are the signatures of Shift, RightShift, and LeftShift. PROCEDURE Shift (x: T; n: INTEGER): T; For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n > 0 is like multiplying by 2^(n) PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; = Shift (x, n) PROCEDURE RightShift (x: T; n: [0..Size-1]): T; = Shift (x, -n) Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 2 Mar 2010, at 14:44, Tony Hosking wrote: > Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > > So, in fact the run-time error is correct! 630 is not in the range [0..63]. > > On 2 Mar 2010, at 13:54, Tony Hosking wrote: > >> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >> >> On 2 Mar 2010, at 09:04, Jay K wrote: >> >>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >>> >>> >>> I'm guessing the warning is all you're supposed to get here? >>> >>> >>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >>> --- building in NT386 --- >>> new source -> compiling Main.m3 >>> "..\Main.m3", line 894: warning: value out of range >>> >>> *** >>> *** runtime error: >>> *** An enumeration or subrange value was out of range. >>> *** file "..\src\TWordN.m3", line 129 >>> *** >>> Stack trace: >>> FP PC Procedure >>> --------- --------- ------------------------------- >>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >>> nWord\Shift.mg >>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >>> ......... ......... ... more frames ... >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 2 22:09:54 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 2 Mar 2010 21:09:54 +0000 Subject: [M3devel] overshifting? In-Reply-To: <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu> References: , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu> Message-ID: I fixed it. It falls back to generating slower code, which is then not hit because it guaranteeably hits a runtime error ahead of it. Perhaps it should just issue a breakpoint there as the code is not reachable. e.g. in C we have: __declspec(noreturn) abort(); void F1() { abort(); /* breakpoint here */ } Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? - Jay From: hosking at cs.purdue.edu Date: Tue, 2 Mar 2010 14:50:27 -0500 To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com; jay.krell at cornell.edu Subject: Re: [M3devel] overshifting? Oh, yes, of course your backend should not blow up on this. On 2 Mar 2010, at 14:46, Tony Hosking wrote: P.S. Here are the signatures of Shift, RightShift, and LeftShift. PROCEDURE Shift (x: T; n: INTEGER): T; For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n > 0 is like multiplying by 2^(n)PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; = Shift (x, n)PROCEDURE RightShift (x: T; n: [0..Size-1]): T; = Shift (x, -n) Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 2 Mar 2010, at 14:44, Tony Hosking wrote: Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. So, in fact the run-time error is correct! 630 is not in the range [0..63]. On 2 Mar 2010, at 13:54, Tony Hosking wrote: I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. On 2 Mar 2010, at 09:04, Jay K wrote: PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); I'm guessing the warning is all you're supposed to get here? C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 --- building in NT386 --- new source -> compiling Main.m3 "..\Main.m3", line 894: warning: value out of range *** *** runtime error: *** An enumeration or subrange value was out of range. *** file "..\src\TWordN.m3", line 129 *** Stack trace: FP PC Procedure --------- --------- ------------------------------- 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti nWord\Shift.mg 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 ......... ......... ... more frames ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 3 00:53:46 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 2 Mar 2010 23:53:46 +0000 Subject: [M3devel] overshifting? In-Reply-To: References: , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, Message-ID: The front end shouldn't bother asking the backend to shift by overlarge constants, eh? Granted, it could be from Shift(a, LAST(INTEGER) + ....); so the frontend can't always know, even if the backend thinks it does. I need to try some things there -- code that produces a large shift via overflowing constants. Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: PROCEDURE Crash() BEGIN EVAL VAL(2, BOOLEAN); END Crash. mostly these are warnings at compile, runtime errors. But some of these things are now barred by m3back, errors, e.g. EVAL LAST(INTEGER) + 1; Maybe I do need that warning callback after all? - Jay ________________________________ > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Tue, 2 Mar 2010 21:09:54 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] overshifting? > > > > > > > > > I fixed it. It falls back to generating slower code, which is then not hit because > > it guaranteeably hits a runtime error ahead of it. > > Perhaps it should just issue a breakpoint there as the code is not reachable. > > e.g. in C we have: > > > > __declspec(noreturn) abort(); > > > > void F1() > > { > abort(); > > /* breakpoint here */ > > } > > > > Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? > > > > - Jay > > > ________________________________ > > From: hosking at cs.purdue.edu > Date: Tue, 2 Mar 2010 14:50:27 -0500 > To: hosking at cs.purdue.edu > CC: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: Re: [M3devel] overshifting? > > > Oh, yes, of course your backend should not blow up on this. > > > > > > > On 2 Mar 2010, at 14:46, Tony Hosking wrote: > > > > P.S. Here are the signatures of Shift, RightShift, and LeftShift. > > > > PROCEDURE Shift (x: T; n: INTEGER): T; > > For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) > > PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > > = Shift (x, n) > > PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > > = Shift (x, -n) > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue University > > 305 N. University Street | West Lafayette | IN 47907 | USA > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > On 2 Mar 2010, at 14:44, Tony Hosking wrote: > > > > > > Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > > > So, in fact the run-time error is correct! 630 is not in the range [0..63]. > > > > On 2 Mar 2010, at 13:54, Tony Hosking wrote: > > > > > I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > > > > On 2 Mar 2010, at 09:04, Jay K wrote: > > > > PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > > > I'm guessing the warning is all you're supposed to get here? > > > C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > --- building in NT386 --- > new source -> compiling Main.m3 > "..\Main.m3", line 894: warning: value out of range > > *** > *** runtime error: > *** An enumeration or subrange value was out of range. > *** file "..\src\TWordN.m3", line 129 > *** > Stack trace: > FP PC Procedure > --------- --------- ------------------------------- > 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > nWord\Shift.mg > 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > ......... ......... ... more frames ... > > > > > From hosking at cs.purdue.edu Wed Mar 3 04:29:47 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 22:29:47 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, Message-ID: The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. On 2 Mar 2010, at 18:53, Jay K wrote: > > The front end shouldn't bother asking the backend to shift by overlarge constants, eh? > Granted, it could be from > Shift(a, LAST(INTEGER) + ....); > > so the frontend can't always know, even if the backend thinks it does. > I need to try some things there -- code that produces a large shift via overflowing constants. > > Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: > > PROCEDURE Crash() > BEGIN > EVAL VAL(2, BOOLEAN); > END Crash. > > > mostly these are warnings at compile, runtime errors. > But some of these things are now barred by m3back, errors, e.g. > > EVAL LAST(INTEGER) + 1; > > Maybe I do need that warning callback after all? > > > - Jay > > > ________________________________ >> From: jay.krell at cornell.edu >> To: hosking at cs.purdue.edu >> Date: Tue, 2 Mar 2010 21:09:54 +0000 >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] overshifting? >> >> >> >> >> >> >> >> >> I fixed it. It falls back to generating slower code, which is then not hit because >> >> it guaranteeably hits a runtime error ahead of it. >> >> Perhaps it should just issue a breakpoint there as the code is not reachable. >> >> e.g. in C we have: >> >> >> >> __declspec(noreturn) abort(); >> >> >> >> void F1() >> >> { >> abort(); >> >> /* breakpoint here */ >> >> } >> >> >> >> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? >> >> >> >> - Jay >> >> >> ________________________________ >> >> From: hosking at cs.purdue.edu >> Date: Tue, 2 Mar 2010 14:50:27 -0500 >> To: hosking at cs.purdue.edu >> CC: m3devel at elegosoft.com; jay.krell at cornell.edu >> Subject: Re: [M3devel] overshifting? >> >> >> Oh, yes, of course your backend should not blow up on this. >> >> >> >> >> >> >> On 2 Mar 2010, at 14:46, Tony Hosking wrote: >> >> >> >> P.S. Here are the signatures of Shift, RightShift, and LeftShift. >> >> >> >> PROCEDURE Shift (x: T; n: INTEGER): T; >> >> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) >> >> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; >> >> = Shift (x, n) >> >> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; >> >> = Shift (x, -n) >> >> >> >> >> >> Antony Hosking | Associate Professor | Computer Science | Purdue University >> >> 305 N. University Street | West Lafayette | IN 47907 | USA >> >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> >> >> On 2 Mar 2010, at 14:44, Tony Hosking wrote: >> >> >> >> >> >> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. >> >> >> So, in fact the run-time error is correct! 630 is not in the range [0..63]. >> >> >> >> On 2 Mar 2010, at 13:54, Tony Hosking wrote: >> >> >> >> >> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >> >> >> >> On 2 Mar 2010, at 09:04, Jay K wrote: >> >> >> >> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >> >> >> I'm guessing the warning is all you're supposed to get here? >> >> >> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >> --- building in NT386 --- >> new source -> compiling Main.m3 >> "..\Main.m3", line 894: warning: value out of range >> >> *** >> *** runtime error: >> *** An enumeration or subrange value was out of range. >> *** file "..\src\TWordN.m3", line 129 >> *** >> Stack trace: >> FP PC Procedure >> --------- --------- ------------------------------- >> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >> nWord\Shift.mg >> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >> ......... ......... ... more frames ... >> >> >> >> >> From jay.krell at cornell.edu Wed Mar 3 05:21:02 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 3 Mar 2010 04:21:02 +0000 Subject: [M3devel] overshifting? In-Reply-To: References: , ,,<9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, ,,<72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, ,,<48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , Message-ID: The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. Doesn't that seem wrong in some way? front end should warn and then backend can be quiet? Why does the backend know more than frontend? Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? - Jay ---------------------------------------- > From: hosking at cs.purdue.edu > Date: Tue, 2 Mar 2010 22:29:47 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] overshifting? > > The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. > > On 2 Mar 2010, at 18:53, Jay K wrote: > >> >> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? >> Granted, it could be from >> Shift(a, LAST(INTEGER) + ....); >> >> so the frontend can't always know, even if the backend thinks it does. >> I need to try some things there -- code that produces a large shift via overflowing constants. >> >> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: >> >> PROCEDURE Crash() >> BEGIN >> EVAL VAL(2, BOOLEAN); >> END Crash. >> >> >> mostly these are warnings at compile, runtime errors. >> But some of these things are now barred by m3back, errors, e.g. >> >> EVAL LAST(INTEGER) + 1; >> >> Maybe I do need that warning callback after all? >> >> >> - Jay >> >> >> ________________________________ >>> From: jay.krell at cornell.edu >>> To: hosking at cs.purdue.edu >>> Date: Tue, 2 Mar 2010 21:09:54 +0000 >>> CC: m3devel at elegosoft.com >>> Subject: Re: [M3devel] overshifting? >>> >>> >>> >>> >>> >>> >>> >>> >>> I fixed it. It falls back to generating slower code, which is then not hit because >>> >>> it guaranteeably hits a runtime error ahead of it. >>> >>> Perhaps it should just issue a breakpoint there as the code is not reachable. >>> >>> e.g. in C we have: >>> >>> >>> >>> __declspec(noreturn) abort(); >>> >>> >>> >>> void F1() >>> >>> { >>> abort(); >>> >>> /* breakpoint here */ >>> >>> } >>> >>> >>> >>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? >>> >>> >>> >>> - Jay >>> >>> >>> ________________________________ >>> >>> From: hosking at cs.purdue.edu >>> Date: Tue, 2 Mar 2010 14:50:27 -0500 >>> To: hosking at cs.purdue.edu >>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu >>> Subject: Re: [M3devel] overshifting? >>> >>> >>> Oh, yes, of course your backend should not blow up on this. >>> >>> >>> >>> >>> >>> >>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: >>> >>> >>> >>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. >>> >>> >>> >>> PROCEDURE Shift (x: T; n: INTEGER): T; >>> >>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) >>> >>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; >>> >>> = Shift (x, n) >>> >>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; >>> >>> = Shift (x, -n) >>> >>> >>> >>> >>> >>> Antony Hosking | Associate Professor | Computer Science | Purdue University >>> >>> 305 N. University Street | West Lafayette | IN 47907 | USA >>> >>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>> >>> >>> >>> >>> >>> >>> >>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: >>> >>> >>> >>> >>> >>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. >>> >>> >>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. >>> >>> >>> >>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: >>> >>> >>> >>> >>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >>> >>> >>> >>> On 2 Mar 2010, at 09:04, Jay K wrote: >>> >>> >>> >>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >>> >>> >>> I'm guessing the warning is all you're supposed to get here? >>> >>> >>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >>> --- building in NT386 --- >>> new source -> compiling Main.m3 >>> "..\Main.m3", line 894: warning: value out of range >>> >>> *** >>> *** runtime error: >>> *** An enumeration or subrange value was out of range. >>> *** file "..\src\TWordN.m3", line 129 >>> *** >>> Stack trace: >>> FP PC Procedure >>> --------- --------- ------------------------------- >>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >>> nWord\Shift.mg >>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >>> ......... ......... ... more frames ... >>> >>> >>> >>> >>> > From jay.krell at cornell.edu Wed Mar 3 05:23:19 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 3 Mar 2010 04:23:19 +0000 Subject: [M3devel] overshifting? In-Reply-To: References: , , , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , , , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , , , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , Message-ID: > You cannot optimise away operations that may cause a run-time error. We don't. We used to. In many cases we absolutely know if there will be overflow or not. If not, optimize. If so, currently, error. But I think maybe it should warn and just call the runtime error functions. Maybe. And I think this is all portable and can be in the frontend. I'm repeating myself now. - Jay ---------------------------------------- > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] overshifting? > Date: Wed, 3 Mar 2010 04:21:02 +0000 > > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > Doesn't that seem wrong in some way? > front end should warn and then backend can be quiet? > Why does the backend know more than frontend? > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > - Jay > > > ---------------------------------------- >> From: hosking at cs.purdue.edu >> Date: Tue, 2 Mar 2010 22:29:47 -0500 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] overshifting? >> >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. >> >> On 2 Mar 2010, at 18:53, Jay K wrote: >> >>> >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? >>> Granted, it could be from >>> Shift(a, LAST(INTEGER) + ....); >>> >>> so the frontend can't always know, even if the backend thinks it does. >>> I need to try some things there -- code that produces a large shift via overflowing constants. >>> >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: >>> >>> PROCEDURE Crash() >>> BEGIN >>> EVAL VAL(2, BOOLEAN); >>> END Crash. >>> >>> >>> mostly these are warnings at compile, runtime errors. >>> But some of these things are now barred by m3back, errors, e.g. >>> >>> EVAL LAST(INTEGER) + 1; >>> >>> Maybe I do need that warning callback after all? >>> >>> >>> - Jay >>> >>> >>> ________________________________ >>>> From: jay.krell at cornell.edu >>>> To: hosking at cs.purdue.edu >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 >>>> CC: m3devel at elegosoft.com >>>> Subject: Re: [M3devel] overshifting? >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> I fixed it. It falls back to generating slower code, which is then not hit because >>>> >>>> it guaranteeably hits a runtime error ahead of it. >>>> >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. >>>> >>>> e.g. in C we have: >>>> >>>> >>>> >>>> __declspec(noreturn) abort(); >>>> >>>> >>>> >>>> void F1() >>>> >>>> { >>>> abort(); >>>> >>>> /* breakpoint here */ >>>> >>>> } >>>> >>>> >>>> >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? >>>> >>>> >>>> >>>> - Jay >>>> >>>> >>>> ________________________________ >>>> >>>> From: hosking at cs.purdue.edu >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 >>>> To: hosking at cs.purdue.edu >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu >>>> Subject: Re: [M3devel] overshifting? >>>> >>>> >>>> Oh, yes, of course your backend should not blow up on this. >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: >>>> >>>> >>>> >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. >>>> >>>> >>>> >>>> PROCEDURE Shift (x: T; n: INTEGER): T; >>>> >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) >>>> >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; >>>> >>>> = Shift (x, n) >>>> >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; >>>> >>>> = Shift (x, -n) >>>> >>>> >>>> >>>> >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University >>>> >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: >>>> >>>> >>>> >>>> >>>> >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. >>>> >>>> >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. >>>> >>>> >>>> >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: >>>> >>>> >>>> >>>> >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >>>> >>>> >>>> >>>> On 2 Mar 2010, at 09:04, Jay K wrote: >>>> >>>> >>>> >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >>>> >>>> >>>> I'm guessing the warning is all you're supposed to get here? >>>> >>>> >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >>>> --- building in NT386 --- >>>> new source -> compiling Main.m3 >>>> "..\Main.m3", line 894: warning: value out of range >>>> >>>> *** >>>> *** runtime error: >>>> *** An enumeration or subrange value was out of range. >>>> *** file "..\src\TWordN.m3", line 129 >>>> *** >>>> Stack trace: >>>> FP PC Procedure >>>> --------- --------- ------------------------------- >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >>>> nWord\Shift.mg >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >>>> ......... ......... ... more frames ... >>>> >>>> >>>> >>>> >>>> >> From hosking at cs.purdue.edu Wed Mar 3 05:44:22 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 23:44:22 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: , , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , Message-ID: <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu> The front-end does fold constants where it can prove no overflow will occur. I don't think I understand any of the rest of what you are saying. On 2 Mar 2010, at 23:21, Jay K wrote: > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > Doesn't that seem wrong in some way? > front end should warn and then backend can be quiet? > Why does the backend know more than frontend? > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > - Jay > > > ---------------------------------------- >> From: hosking at cs.purdue.edu >> Date: Tue, 2 Mar 2010 22:29:47 -0500 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] overshifting? >> >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. >> >> On 2 Mar 2010, at 18:53, Jay K wrote: >> >>> >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? >>> Granted, it could be from >>> Shift(a, LAST(INTEGER) + ....); >>> >>> so the frontend can't always know, even if the backend thinks it does. >>> I need to try some things there -- code that produces a large shift via overflowing constants. >>> >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: >>> >>> PROCEDURE Crash() >>> BEGIN >>> EVAL VAL(2, BOOLEAN); >>> END Crash. >>> >>> >>> mostly these are warnings at compile, runtime errors. >>> But some of these things are now barred by m3back, errors, e.g. >>> >>> EVAL LAST(INTEGER) + 1; >>> >>> Maybe I do need that warning callback after all? >>> >>> >>> - Jay >>> >>> >>> ________________________________ >>>> From: jay.krell at cornell.edu >>>> To: hosking at cs.purdue.edu >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 >>>> CC: m3devel at elegosoft.com >>>> Subject: Re: [M3devel] overshifting? >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> I fixed it. It falls back to generating slower code, which is then not hit because >>>> >>>> it guaranteeably hits a runtime error ahead of it. >>>> >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. >>>> >>>> e.g. in C we have: >>>> >>>> >>>> >>>> __declspec(noreturn) abort(); >>>> >>>> >>>> >>>> void F1() >>>> >>>> { >>>> abort(); >>>> >>>> /* breakpoint here */ >>>> >>>> } >>>> >>>> >>>> >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? >>>> >>>> >>>> >>>> - Jay >>>> >>>> >>>> ________________________________ >>>> >>>> From: hosking at cs.purdue.edu >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 >>>> To: hosking at cs.purdue.edu >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu >>>> Subject: Re: [M3devel] overshifting? >>>> >>>> >>>> Oh, yes, of course your backend should not blow up on this. >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: >>>> >>>> >>>> >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. >>>> >>>> >>>> >>>> PROCEDURE Shift (x: T; n: INTEGER): T; >>>> >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) >>>> >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; >>>> >>>> = Shift (x, n) >>>> >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; >>>> >>>> = Shift (x, -n) >>>> >>>> >>>> >>>> >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University >>>> >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: >>>> >>>> >>>> >>>> >>>> >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. >>>> >>>> >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. >>>> >>>> >>>> >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: >>>> >>>> >>>> >>>> >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >>>> >>>> >>>> >>>> On 2 Mar 2010, at 09:04, Jay K wrote: >>>> >>>> >>>> >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >>>> >>>> >>>> I'm guessing the warning is all you're supposed to get here? >>>> >>>> >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >>>> --- building in NT386 --- >>>> new source -> compiling Main.m3 >>>> "..\Main.m3", line 894: warning: value out of range >>>> >>>> *** >>>> *** runtime error: >>>> *** An enumeration or subrange value was out of range. >>>> *** file "..\src\TWordN.m3", line 129 >>>> *** >>>> Stack trace: >>>> FP PC Procedure >>>> --------- --------- ------------------------------- >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >>>> nWord\Shift.mg >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >>>> ......... ......... ... more frames ... >>>> >>>> >>>> >>>> >>>> >> From jay.krell at cornell.edu Wed Mar 3 07:00:46 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 3 Mar 2010 06:00:46 +0000 Subject: [M3devel] overshifting? In-Reply-To: <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu> References: , , ,,<9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , ,,<72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , ,,<48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , ,,<170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , , , , , , <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu> Message-ID: Why does Word.LeftShift(x , 100) even get to the backend? It should just generate the code to issue a runtime error? (even if x could be zero? I think so.) Why doesn't front warn for -FIRST(INTEGER)? It should, right? But still generate code to compute it. Are these just oversights? - Jay > From: hosking at cs.purdue.edu > Date: Tue, 2 Mar 2010 23:44:22 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] overshifting? > > The front-end does fold constants where it can prove no overflow will occur. I don't think I understand any of the rest of what you are saying. > > On 2 Mar 2010, at 23:21, Jay K wrote: > > > > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > > Doesn't that seem wrong in some way? > > front end should warn and then backend can be quiet? > > Why does the backend know more than frontend? > > > > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > > > > > - Jay > > > > > > ---------------------------------------- > >> From: hosking at cs.purdue.edu > >> Date: Tue, 2 Mar 2010 22:29:47 -0500 > >> To: jay.krell at cornell.edu > >> CC: m3devel at elegosoft.com > >> Subject: Re: [M3devel] overshifting? > >> > >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. > >> > >> On 2 Mar 2010, at 18:53, Jay K wrote: > >> > >>> > >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? > >>> Granted, it could be from > >>> Shift(a, LAST(INTEGER) + ....); > >>> > >>> so the frontend can't always know, even if the backend thinks it does. > >>> I need to try some things there -- code that produces a large shift via overflowing constants. > >>> > >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: > >>> > >>> PROCEDURE Crash() > >>> BEGIN > >>> EVAL VAL(2, BOOLEAN); > >>> END Crash. > >>> > >>> > >>> mostly these are warnings at compile, runtime errors. > >>> But some of these things are now barred by m3back, errors, e.g. > >>> > >>> EVAL LAST(INTEGER) + 1; > >>> > >>> Maybe I do need that warning callback after all? > >>> > >>> > >>> - Jay > >>> > >>> > >>> ________________________________ > >>>> From: jay.krell at cornell.edu > >>>> To: hosking at cs.purdue.edu > >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 > >>>> CC: m3devel at elegosoft.com > >>>> Subject: Re: [M3devel] overshifting? > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> I fixed it. It falls back to generating slower code, which is then not hit because > >>>> > >>>> it guaranteeably hits a runtime error ahead of it. > >>>> > >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. > >>>> > >>>> e.g. in C we have: > >>>> > >>>> > >>>> > >>>> __declspec(noreturn) abort(); > >>>> > >>>> > >>>> > >>>> void F1() > >>>> > >>>> { > >>>> abort(); > >>>> > >>>> /* breakpoint here */ > >>>> > >>>> } > >>>> > >>>> > >>>> > >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? > >>>> > >>>> > >>>> > >>>> - Jay > >>>> > >>>> > >>>> ________________________________ > >>>> > >>>> From: hosking at cs.purdue.edu > >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 > >>>> To: hosking at cs.purdue.edu > >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu > >>>> Subject: Re: [M3devel] overshifting? > >>>> > >>>> > >>>> Oh, yes, of course your backend should not blow up on this. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. > >>>> > >>>> > >>>> > >>>> PROCEDURE Shift (x: T; n: INTEGER): T; > >>>> > >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) > >>>> > >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > >>>> > >>>> = Shift (x, n) > >>>> > >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > >>>> > >>>> = Shift (x, -n) > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University > >>>> > >>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>> > >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > >>>> > >>>> > >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> > >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 09:04, Jay K wrote: > >>>> > >>>> > >>>> > >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > >>>> > >>>> > >>>> I'm guessing the warning is all you're supposed to get here? > >>>> > >>>> > >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > >>>> --- building in NT386 --- > >>>> new source -> compiling Main.m3 > >>>> "..\Main.m3", line 894: warning: value out of range > >>>> > >>>> *** > >>>> *** runtime error: > >>>> *** An enumeration or subrange value was out of range. > >>>> *** file "..\src\TWordN.m3", line 129 > >>>> *** > >>>> Stack trace: > >>>> FP PC Procedure > >>>> --------- --------- ------------------------------- > >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > >>>> nWord\Shift.mg > >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > >>>> ......... ......... ... more frames ... > >>>> > >>>> > >>>> > >>>> > >>>> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 3 16:54:00 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 3 Mar 2010 10:54:00 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: , , , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , , , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , , , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , , , , , , <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu> Message-ID: On 3 Mar 2010, at 01:00, Jay K wrote: > Why does Word.LeftShift(x , 100) even get to the backend? It should just generate the code to issue a runtime error? (even if x could be zero? I think so.) Because the language spec says that a range error (100 does not fit in the range [0..Word.Size-1]) is a run-time error. Note that the code generator's M3CG_Ops don't specify ranges. The range check is in the *interface* for Word. The m3middle and backend M3CG_Ops are intended to be more general. So, M3CG_Ops.shift_left and shift_right are permitted to take an argument that is not range-checked. > Why doesn't front warn for -FIRST(INTEGER)? It should, right? This is independent of the range check above. This is an expression that may or may not evaluate depending on the run-time defaults (checking for overflow or not). The compiler must be conservative and let the run-time influence whether overflow checking is in place or not. Overflow checking is a run-time option. The compiler must defer to the run-time. > But still generate code to compute it. > Are these just oversights? Nope. Intentional. > > - Jay > > > From: hosking at cs.purdue.edu > > Date: Tue, 2 Mar 2010 23:44:22 -0500 > > To: jay.krell at cornell.edu > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] overshifting? > > > > The front-end does fold constants where it can prove no overflow will occur. I don't think I understand any of the rest of what you are saying. > > > > On 2 Mar 2010, at 23:21, Jay K wrote: > > > > > > > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > > > > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > > > > > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > > > > > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > > > Doesn't that seem wrong in some way? > > > front end should warn and then backend can be quiet? > > > Why does the backend know more than frontend? > > > > > > > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > > > > > > > > > - Jay > > > > > > > > > ---------------------------------------- > > >> From: hosking at cs.purdue.edu > > >> Date: Tue, 2 Mar 2010 22:29:47 -0500 > > >> To: jay.krell at cornell.edu > > >> CC: m3devel at elegosoft.com > > >> Subject: Re: [M3devel] overshifting? > > >> > > >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. > > >> > > >> On 2 Mar 2010, at 18:53, Jay K wrote: > > >> > > >>> > > >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? > > >>> Granted, it could be from > > >>> Shift(a, LAST(INTEGER) + ....); > > >>> > > >>> so the frontend can't always know, even if the backend thinks it does. > > >>> I need to try some things there -- code that produces a large shift via overflowing constants. > > >>> > > >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: > > >>> > > >>> PROCEDURE Crash() > > >>> BEGIN > > >>> EVAL VAL(2, BOOLEAN); > > >>> END Crash. > > >>> > > >>> > > >>> mostly these are warnings at compile, runtime errors. > > >>> But some of these things are now barred by m3back, errors, e.g. > > >>> > > >>> EVAL LAST(INTEGER) + 1; > > >>> > > >>> Maybe I do need that warning callback after all? > > >>> > > >>> > > >>> - Jay > > >>> > > >>> > > >>> ________________________________ > > >>>> From: jay.krell at cornell.edu > > >>>> To: hosking at cs.purdue.edu > > >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 > > >>>> CC: m3devel at elegosoft.com > > >>>> Subject: Re: [M3devel] overshifting? > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> I fixed it. It falls back to generating slower code, which is then not hit because > > >>>> > > >>>> it guaranteeably hits a runtime error ahead of it. > > >>>> > > >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. > > >>>> > > >>>> e.g. in C we have: > > >>>> > > >>>> > > >>>> > > >>>> __declspec(noreturn) abort(); > > >>>> > > >>>> > > >>>> > > >>>> void F1() > > >>>> > > >>>> { > > >>>> abort(); > > >>>> > > >>>> /* breakpoint here */ > > >>>> > > >>>> } > > >>>> > > >>>> > > >>>> > > >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? > > >>>> > > >>>> > > >>>> > > >>>> - Jay > > >>>> > > >>>> > > >>>> ________________________________ > > >>>> > > >>>> From: hosking at cs.purdue.edu > > >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 > > >>>> To: hosking at cs.purdue.edu > > >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu > > >>>> Subject: Re: [M3devel] overshifting? > > >>>> > > >>>> > > >>>> Oh, yes, of course your backend should not blow up on this. > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. > > >>>> > > >>>> > > >>>> > > >>>> PROCEDURE Shift (x: T; n: INTEGER): T; > > >>>> > > >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) > > >>>> > > >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > > >>>> > > >>>> = Shift (x, n) > > >>>> > > >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > > >>>> > > >>>> = Shift (x, -n) > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University > > >>>> > > >>>> 305 N. University Street | West Lafayette | IN 47907 | USA > > >>>> > > >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > > >>>> > > >>>> > > >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 09:04, Jay K wrote: > > >>>> > > >>>> > > >>>> > > >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > > >>>> > > >>>> > > >>>> I'm guessing the warning is all you're supposed to get here? > > >>>> > > >>>> > > >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > > >>>> --- building in NT386 --- > > >>>> new source -> compiling Main.m3 > > >>>> "..\Main.m3", line 894: warning: value out of range > > >>>> > > >>>> *** > > >>>> *** runtime error: > > >>>> *** An enumeration or subrange value was out of range. > > >>>> *** file "..\src\TWordN.m3", line 129 > > >>>> *** > > >>>> Stack trace: > > >>>> FP PC Procedure > > >>>> --------- --------- ------------------------------- > > >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > > >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > > >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > > >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > > >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > > >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > > >>>> nWord\Shift.mg > > >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > > >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > > >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > > >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > > >>>> ......... ......... ... more frames ... > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 4 00:52:54 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 3 Mar 2010 23:52:54 +0000 Subject: [M3devel] overshifting? In-Reply-To: References: , , , ,,<9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , , ,,<72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , , ,,<48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , ,,<170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , ,,, ,,, , , , , , , <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu>, , Message-ID: Tony you aren't understanding me. I'm not suggesting to optimize away runtime errors. I understand that part of the spec. It has been referenced multiple times recently. I'm saying that the front end knows often/always without a doubt if an operation will overflow or not. If the operation will not overflow, it can do the work itself. From what you say, it already does. (I'm going to confuse "overflow" and "subrange error" here briefly.) If the operation will overflow, it can just generate code to issue the error. Now, for most operations, it doesn't know how the target responds to overflow, so the second statement isn't quite right. However, if the operation absolutely will overflow, the frontend should warn. It at least misses some cases, e.g. -FIRST(INTEGER). Granted, on a one's complement system, it won't. But it should warn for the sake of all two's complement systems, which is all of them. Let's amend the previous: If the operation absolutely will overflow, the front end should warn, and then just ask CG to generate the usual code, as if the operation wouldn't overflow. I think it misses some warnings currently. I think it's easy to fix and I'll look into it. Basically whenever the TInt operations fail, should warn. Now, let's somewhat replace "overflow" with "subrange error". LeftShift(x, 100) acts the same on all targets. It isn't overflow, it is subrange error. In this case, I assert, the front end should warn, and it shouldn't bother asking the backend to generate code to the shift. -Jay From: hosking at cs.purdue.edu Date: Wed, 3 Mar 2010 10:54:00 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] overshifting? On 3 Mar 2010, at 01:00, Jay K wrote: Why does Word.LeftShift(x , 100) even get to the backend? It should just generate the code to issue a runtime error? (even if x could be zero? I think so.) Because the language spec says that a range error (100 does not fit in the range [0..Word.Size-1]) is a run-time error. Note that the code generator's M3CG_Ops don't specify ranges. The range check is in the *interface* for Word. The m3middle and backend M3CG_Ops are intended to be more general. So, M3CG_Ops.shift_left and shift_right are permitted to take an argument that is not range-checked. Why doesn't front warn for -FIRST(INTEGER)? It should, right? This is independent of the range check above. This is an expression that may or may not evaluate depending on the run-time defaults (checking for overflow or not). The compiler must be conservative and let the run-time influence whether overflow checking is in place or not. Overflow checking is a run-time option. The compiler must defer to the run-time. But still generate code to compute it. Are these just oversights? Nope. Intentional. - Jay > From: hosking at cs.purdue.edu > Date: Tue, 2 Mar 2010 23:44:22 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] overshifting? > > The front-end does fold constants where it can prove no overflow will occur. I don't think I understand any of the rest of what you are saying. > > On 2 Mar 2010, at 23:21, Jay K wrote: > > > > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > > Doesn't that seem wrong in some way? > > front end should warn and then backend can be quiet? > > Why does the backend know more than frontend? > > > > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > > > > > - Jay > > > > > > ---------------------------------------- > >> From: hosking at cs.purdue.edu > >> Date: Tue, 2 Mar 2010 22:29:47 -0500 > >> To: jay.krell at cornell.edu > >> CC: m3devel at elegosoft.com > >> Subject: Re: [M3devel] overshifting? > >> > >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. > >> > >> On 2 Mar 2010, at 18:53, Jay K wrote: > >> > >>> > >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? > >>> Granted, it could be from > >>> Shift(a, LAST(INTEGER) + ....); > >>> > >>> so the frontend can't always know, even if the backend thinks it does. > >>> I need to try some things there -- code that produces a large shift via overflowing constants. > >>> > >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: > >>> > >>> PROCEDURE Crash() > >>> BEGIN > >>> EVAL VAL(2, BOOLEAN); > >>> END Crash. > >>> > >>> > >>> mostly these are warnings at compile, runtime errors. > >>> But some of these things are now barred by m3back, errors, e.g. > >>> > >>> EVAL LAST(INTEGER) + 1; > >>> > >>> Maybe I do need that warning callback after all? > >>> > >>> > >>> - Jay > >>> > >>> > >>> ________________________________ > >>>> From: jay.krell at cornell.edu > >>>> To: hosking at cs.purdue.edu > >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 > >>>> CC: m3devel at elegosoft.com > >>>> Subject: Re: [M3devel] overshifting? > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> I fixed it. It falls back to generating slower code, which is then not hit because > >>>> > >>>> it guaranteeably hits a runtime error ahead of it. > >>>> > >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. > >>>> > >>>> e.g. in C we have: > >>>> > >>>> > >>>> > >>>> __declspec(noreturn) abort(); > >>>> > >>>> > >>>> > >>>> void F1() > >>>> > >>>> { > >>>> abort(); > >>>> > >>>> /* breakpoint here */ > >>>> > >>>> } > >>>> > >>>> > >>>> > >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? > >>>> > >>>> > >>>> > >>>> - Jay > >>>> > >>>> > >>>> ________________________________ > >>>> > >>>> From: hosking at cs.purdue.edu > >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 > >>>> To: hosking at cs.purdue.edu > >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu > >>>> Subject: Re: [M3devel] overshifting? > >>>> > >>>> > >>>> Oh, yes, of course your backend should not blow up on this. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. > >>>> > >>>> > >>>> > >>>> PROCEDURE Shift (x: T; n: INTEGER): T; > >>>> > >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) > >>>> > >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > >>>> > >>>> = Shift (x, n) > >>>> > >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > >>>> > >>>> = Shift (x, -n) > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University > >>>> > >>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>> > >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > >>>> > >>>> > >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> > >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 09:04, Jay K wrote: > >>>> > >>>> > >>>> > >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > >>>> > >>>> > >>>> I'm guessing the warning is all you're supposed to get here? > >>>> > >>>> > >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > >>>> --- building in NT386 --- > >>>> new source -> compiling Main.m3 > >>>> "..\Main.m3", line 894: warning: value out of range > >>>> > >>>> *** > >>>> *** runtime error: > >>>> *** An enumeration or subrange value was out of range. > >>>> *** file "..\src\TWordN.m3", line 129 > >>>> *** > >>>> Stack trace: > >>>> FP PC Procedure > >>>> --------- --------- ------------------------------- > >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > >>>> nWord\Shift.mg > >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > >>>> ......... ......... ... more frames ... > >>>> > >>>> > >>>> > >>>> > >>>> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 4 01:45:07 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 4 Mar 2010 00:45:07 +0000 Subject: [M3devel] set operations parse.c? Message-ID: The flag to setop is maybe yucky, but I think static void m3cg_set_eq (void) { m3cg_set_eq_or_ne(EQ_EXPR); } static void m3cg_set_ne (void) { m3cg_set_eq_or_ne(NE_EXPR); } is still good. There are two functions *very* similar in functionality that vary in only one token. The counterpoint would be, perhaps, if one is proficient enough in maintaining parse.c, that the two functions are small and one could write them up in a flash without referring to examples. One needs that level of proficiency to make more significant changes anyway. (I'll be removing the set_singleton/member functions shortly, assuming reasonable codegen.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 4 04:20:07 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 3 Mar 2010 22:20:07 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: , , , , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , , , , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , , , , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , , , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , , , , , , , , , , , <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu>, , Message-ID: <4671EB9C-6182-462C-B04F-89428155D530@cs.purdue.edu> On 3 Mar 2010, at 18:52, Jay K wrote: > Tony you aren't understanding me. > > > I'm not suggesting to optimize away runtime errors. > I understand that part of the spec. It has been referenced multiple times recently. > > > I'm saying that the front end knows often/always without a doubt if an operation will overflow or not. > If the operation will not overflow, it can do the work itself. > From what you say, it already does. Correct. > (I'm going to confuse "overflow" and "subrange error" here briefly.) But they are distinct concepts! They cannot be merged. > If the operation will overflow, it can just generate code to issue the error. No! Overflow is a run-time error only if enabled at run-time using FloatMode (or if the target checks for overflow). I think you are not understanding me! > Now, for most operations, it doesn't know how the target responds to overflow, so the second statement isn't quite right. > However, if the operation absolutely will overflow, the frontend should warn. It at least misses some cases, e.g. -FIRST(INTEGER). Granted, on a one's complement system, it won't. But it should warn for the sake of all two's complement systems, which is all of them. > > Let's amend the previous: > If the operation absolutely will overflow, the front end should warn, and then just ask CG to generate the usual code, as if the operation wouldn't overflow. I think it misses some warnings currently. I think it's easy to fix and I'll look into it. Basically whenever the TInt operations fail, should warn. > > > Now, let's somewhat replace "overflow" with "subrange error". > LeftShift(x, 100) > acts the same on all targets. It isn't overflow, it is subrange error. > In this case, I assert, the front end should warn, and it shouldn't bother asking the backend to generate code to the shift. > > > -Jay > > > From: hosking at cs.purdue.edu > Date: Wed, 3 Mar 2010 10:54:00 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] overshifting? > > On 3 Mar 2010, at 01:00, Jay K wrote: > > Why does Word.LeftShift(x , 100) even get to the backend? It should just generate the code to issue a runtime error? (even if x could be zero? I think so.) > > Because the language spec says that a range error (100 does not fit in the range [0..Word.Size-1]) is a run-time error. Note that the code generator's M3CG_Ops don't specify ranges. The range check is in the *interface* for Word. The m3middle and backend M3CG_Ops are intended to be more general. So, M3CG_Ops.shift_left and shift_right are permitted to take an argument that is not range-checked. > > Why doesn't front warn for -FIRST(INTEGER)? It should, right? > > This is independent of the range check above. This is an expression that may or may not evaluate depending on the run-time defaults (checking for overflow or not). The compiler must be conservative and let the run-time influence whether overflow checking is in place or not. > > Overflow checking is a run-time option. The compiler must defer to the run-time. > > But still generate code to compute it. > Are these just oversights? > > Nope. Intentional. > > > - Jay > > > From: hosking at cs.purdue.edu > > Date: Tue, 2 Mar 2010 23:44:22 -0500 > > To: jay.krell at cornell.edu > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] overshifting? > > > > The front-end does fold constants where it can prove no overflow will occur. I don't think I understand any of the rest of what you are saying. > > > > On 2 Mar 2010, at 23:21, Jay K wrote: > > > > > > > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > > > > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > > > > > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > > > > > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > > > Doesn't that seem wrong in some way? > > > front end should warn and then backend can be quiet? > > > Why does the backend know more than frontend? > > > > > > > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > > > > > > > > > - Jay > > > > > > > > > ---------------------------------------- > > >> From: hosking at cs.purdue.edu > > >> Date: Tue, 2 Mar 2010 22:29:47 -0500 > > >> To: jay.krell at cornell.edu > > >> CC: m3devel at elegosoft.com > > >> Subject: Re: [M3devel] overshifting? > > >> > > >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. > > >> > > >> On 2 Mar 2010, at 18:53, Jay K wrote: > > >> > > >>> > > >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? > > >>> Granted, it could be from > > >>> Shift(a, LAST(INTEGER) + ....); > > >>> > > >>> so the frontend can't always know, even if the backend thinks it does. > > >>> I need to try some things there -- code that produces a large shift via overflowing constants. > > >>> > > >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: > > >>> > > >>> PROCEDURE Crash() > > >>> BEGIN > > >>> EVAL VAL(2, BOOLEAN); > > >>> END Crash. > > >>> > > >>> > > >>> mostly these are warnings at compile, runtime errors. > > >>> But some of these things are now barred by m3back, errors, e.g. > > >>> > > >>> EVAL LAST(INTEGER) + 1; > > >>> > > >>> Maybe I do need that warning callback after all? > > >>> > > >>> > > >>> - Jay > > >>> > > >>> > > >>> ________________________________ > > >>>> From: jay.krell at cornell.edu > > >>>> To: hosking at cs.purdue.edu > > >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 > > >>>> CC: m3devel at elegosoft.com > > >>>> Subject: Re: [M3devel] overshifting? > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> I fixed it. It falls back to generating slower code, which is then not hit because > > >>>> > > >>>> it guaranteeably hits a runtime error ahead of it. > > >>>> > > >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. > > >>>> > > >>>> e.g. in C we have: > > >>>> > > >>>> > > >>>> > > >>>> __declspec(noreturn) abort(); > > >>>> > > >>>> > > >>>> > > >>>> void F1() > > >>>> > > >>>> { > > >>>> abort(); > > >>>> > > >>>> /* breakpoint here */ > > >>>> > > >>>> } > > >>>> > > >>>> > > >>>> > > >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? > > >>>> > > >>>> > > >>>> > > >>>> - Jay > > >>>> > > >>>> > > >>>> ________________________________ > > >>>> > > >>>> From: hosking at cs.purdue.edu > > >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 > > >>>> To: hosking at cs.purdue.edu > > >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu > > >>>> Subject: Re: [M3devel] overshifting? > > >>>> > > >>>> > > >>>> Oh, yes, of course your backend should not blow up on this. > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. > > >>>> > > >>>> > > >>>> > > >>>> PROCEDURE Shift (x: T; n: INTEGER): T; > > >>>> > > >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) > > >>>> > > >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > > >>>> > > >>>> = Shift (x, n) > > >>>> > > >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > > >>>> > > >>>> = Shift (x, -n) > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University > > >>>> > > >>>> 305 N. University Street | West Lafayette | IN 47907 | USA > > >>>> > > >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > > >>>> > > >>>> > > >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 09:04, Jay K wrote: > > >>>> > > >>>> > > >>>> > > >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > > >>>> > > >>>> > > >>>> I'm guessing the warning is all you're supposed to get here? > > >>>> > > >>>> > > >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > > >>>> --- building in NT386 --- > > >>>> new source -> compiling Main.m3 > > >>>> "..\Main.m3", line 894: warning: value out of range > > >>>> > > >>>> *** > > >>>> *** runtime error: > > >>>> *** An enumeration or subrange value was out of range. > > >>>> *** file "..\src\TWordN.m3", line 129 > > >>>> *** > > >>>> Stack trace: > > >>>> FP PC Procedure > > >>>> --------- --------- ------------------------------- > > >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > > >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > > >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > > >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > > >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > > >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > > >>>> nWord\Shift.mg > > >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > > >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > > >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > > >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > > >>>> ......... ......... ... more frames ... > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 4 04:27:55 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 3 Mar 2010 22:27:55 -0500 Subject: [M3devel] set operations parse.c? In-Reply-To: References: Message-ID: It obscures the simple intent of the eq/ne operations, and conflates them with other similarly simple set operations. Agreed, the implementations now vary in only one token, but I can read them each in a single page of my editor and validate their correctness without having to trace through multiple calls and track a flag value in context along the way. On 3 Mar 2010, at 19:45, Jay K wrote: > The flag to setop is maybe yucky, but I think > static void m3cg_set_eq (void) { m3cg_set_eq_or_ne(EQ_EXPR); } > static void m3cg_set_ne (void) { m3cg_set_eq_or_ne(NE_EXPR); } > > > is still good. > There are two functions *very* similar in functionality that vary in only one token. > > > The counterpoint would be, perhaps, if one is proficient enough > in maintaining parse.c, that the two functions are small and > one could write them up in a flash without referring to examples. EXACTLY! > One needs that level of proficiency to make more significant changes anyway. > (I'll be removing the set_singleton/member functions shortly, assuming > reasonable codegen.) > > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 4 07:44:41 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 4 Mar 2010 06:44:41 +0000 Subject: [M3devel] overshift/overflow Message-ID: Tony, these two questions seemed to go unanswered: 1) What should this code do: Word.LeftShift(..., 100); (* where 100 >= BITSIZE(INTEGER), and similar for Long.LeftShift and shift >= BITSIZE(LONGINT) *) I think m3front should generate a warning and then either generate the same code as today, or something smaller where it only generates code to issue the runtime error. The warning is missing. "Generating the code same as today" is harder on backends, since they have to check for this condition and handle it somehow, though it doesn't matter much what they do, since the runtime error generation will always run and anything after it never will. 2) What should this code do: EVAL -FIRST(INTEGER); I believe the frontend should issue a warning. And then generate the same code as today. Just the warning is missing. 3) Aside, and not a question. I believe, esp. at this point, the notion that overflow checking is a per-thread setting is a mistake. It has "never" been used, save on a small number of targets. It is too late to foist that change on any code thread-wide. The correct thing to do is introduce different interfaces/modules/types/functions which either always do overflow checking, or, perhaps but less likely, new interfaces/modules/types/functions that are runtime configurable, as INTEGER was originally speced. Even the original spec not so great in my opinion. There are too many things to code correctly, let alone worrying about "this" possibly varying underneath you. The overall system is too much a combination of different code to expect just because one set of code wants integer overflow to be an error, that all the code can deal with that. Better to have separate interfaces/functions/types for the different functionality. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 4 07:55:14 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 4 Mar 2010 06:55:14 +0000 Subject: [M3devel] set operations parse.c? In-Reply-To: References: , Message-ID: Any reuse is too obscuring and not worthwhile if the functions already fit on a page? Huh. By that metric, setop, setop2, m3cg_set_compare aren't needed either. Actually I do find setop vs. setop2 a *little* hard to read. And I hope to soon get rid of two out of its three uses pretty soon (set_singleton, member), maybe tonight. The relationship between eq and ne seems among the strongest, except for all the other comparisons. Actually I'm not sure I "like" a lot here. There is or can be enough similarity in the backends, that the frontend probably should just generate either inline code or calls to portable Modula-3 functions, at least lt/le/gt/ge, and to memcmp for eq/ne. singleton/member I'm a bit on the fence. They can be generated portably inline with pretty small code, though that'd take away the ability of m3back to easily know to use bt/bts instructions. You know, my big "fight" is with "too much code". Anything that the frontend can do pretty darn well, is better than doing it in multiple backends. "More higher level operations" are most useful so a "simpler" backend, such as m3back but not gcc, can more easily generate better code. - Jay Subject: Re: set operations parse.c? From: hosking at cs.purdue.edu Date: Wed, 3 Mar 2010 22:27:55 -0500 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu It obscures the simple intent of the eq/ne operations, and conflates them with other similarly simple set operations. Agreed, the implementations now vary in only one token, but I can read them each in a single page of my editor and validate their correctness without having to trace through multiple calls and track a flag value in context along the way. On 3 Mar 2010, at 19:45, Jay K wrote: The flag to setop is maybe yucky, but I think static void m3cg_set_eq (void) { m3cg_set_eq_or_ne(EQ_EXPR); } static void m3cg_set_ne (void) { m3cg_set_eq_or_ne(NE_EXPR); } is still good. There are two functions *very* similar in functionality that vary in only one token. The counterpoint would be, perhaps, if one is proficient enough in maintaining parse.c, that the two functions are small and one could write them up in a flash without referring to examples. EXACTLY! One needs that level of proficiency to make more significant changes anyway. (I'll be removing the set_singleton/member functions shortly, assuming reasonable codegen.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 4 08:43:31 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 4 Mar 2010 02:43:31 -0500 Subject: [M3devel] set operations parse.c? In-Reply-To: References: , Message-ID: On 4 Mar 2010, at 01:55, Jay K wrote: > Any reuse is too obscuring and not worthwhile if the functions already fit on a page? > Huh. Why hide simple code behind a contorted flag-driven procedure abstraction? > By that metric, setop, setop2, m3cg_set_compare aren't needed either. They don't require calling context to understand what they mean. > Actually I do find setop vs. setop2 a *little* hard to read. ;-) > And I hope to soon get rid of two out of its three uses pretty soon (set_singleton, member), maybe tonight. > > > The relationship between eq and ne seems among the strongest, except for all the other comparisons. > > > Actually I'm not sure I "like" a lot here. > There is or can be enough similarity in the backends, that the frontend probably should just generate either inline code or calls to portable Modula-3 functions, at least lt/le/gt/ge, and to memcmp for eq/ne. Too much work for the front-end. It shouldn't have to worry about such things. > singleton/member I'm a bit on the fence. > They can be generated portably inline with pretty small code, though that'd take away the ability of m3back to easily know to use bt/bts instructions. > > > You know, my big "fight" is with "too much code". Anything that the frontend can do pretty darn well, is better than doing it in multiple backends. "More higher level operations" are most useful so a "simpler" backend, such as m3back but not gcc, can more easily generate better code. > > > - Jay > > > Subject: Re: set operations parse.c? > From: hosking at cs.purdue.edu > Date: Wed, 3 Mar 2010 22:27:55 -0500 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > > It obscures the simple intent of the eq/ne operations, and conflates them with other similarly simple set operations. > > Agreed, the implementations now vary in only one token, but I can read them each in a single page of my editor and validate their correctness without having to trace through multiple calls and track a flag value in context along the way. > > On 3 Mar 2010, at 19:45, Jay K wrote: > > The flag to setop is maybe yucky, but I think > static void m3cg_set_eq (void) { m3cg_set_eq_or_ne(EQ_EXPR); } > static void m3cg_set_ne (void) { m3cg_set_eq_or_ne(NE_EXPR); } > > > is still good. > There are two functions *very* similar in functionality that vary in only one token. > > > The counterpoint would be, perhaps, if one is proficient enough > in maintaining parse.c, that the two functions are small and > one could write them up in a flash without referring to examples. > > EXACTLY! > > One needs that level of proficiency to make more significant changes anyway. > (I'll be removing the set_singleton/member functions shortly, assuming > reasonable codegen.) > > > - Jay > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 4 08:57:26 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 4 Mar 2010 02:57:26 -0500 Subject: [M3devel] overshift/overflow In-Reply-To: References: Message-ID: I already answered these questions. On 4 Mar 2010, at 01:44, Jay K wrote: > Tony, these two questions seemed to go unanswered: > > 1) > > What should this code do: > > > Word.LeftShift(..., 100); (* where 100 >= BITSIZE(INTEGER), > and similar for Long.LeftShift and shift >= BITSIZE(LONGINT) *) > > > I think m3front should generate a warning It does generate a warning. > and then either generate the same code as today, > or something smaller where it only generates code > to issue the runtime error. The code for the shift doesn't know that the warning has been generated, since it is all in the checking of the arguments. > The warning is missing. No, it is generated. > "Generating the code same as today" is harder on backends, > since they have to check for this condition and handle it somehow, > though it doesn't matter much what they do, since the runtime > error generation will always run and anything after it never will. Why can't a backend simply generate code for the large shift, as if it had been a call to Word.Shift(..., 100)? > 2) What should this code do: > > > EVAL -FIRST(INTEGER); > > > I believe the frontend should issue a warning. Why? The front-end does not reason about overflow (except when computing compile-time constants). Overflow is a run-time concept!!!!!!!!!!!!! > And then generate the same code as today. > Just the warning is missing. > > > > 3) Aside, and not a question. > I believe, esp. at this point, the notion that overflow checking is a per-thread setting > is a mistake. It has "never" been used, save on a small number of targets. > It is too late to foist that change on any code thread-wide. > The correct thing to do is introduce different interfaces/modules/types/functions > which either always do overflow checking, or, perhaps but less likely, > new interfaces/modules/types/functions that are runtime configurable, as > INTEGER was originally speced. NOOOOO!!!!!!! That will impose an undue expense on targets where such checking is expensive. Targets where trap handlers can be used to catch overflow will enable it to be turned on via FloatMode. Targets that don't support it efficiently don't have to! > Even the original spec not so great in my opinion. > There are too many things to code correctly, let alone worrying about "this" > possibly varying underneath you. The overall system is too much a combination > of different code to expect just because one set of code wants integer overflow > to be an error, that all the code can deal with that. > Better to have separate interfaces/functions/types for the different functionality. > > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 4 10:00:11 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 4 Mar 2010 09:00:11 +0000 Subject: [M3devel] overshift/overflow In-Reply-To: References: , Message-ID: > Word.LeftShift(..., 100); (* where 100 >= BITSIZE(INTEGER), My mistake, it does generate a warning. It also seems to not generate the code to do the shift, good. I had probably hidden 100 behind a function call, inhibiting the front end's checking. Or more likely behind something else. I have to look into something. > Why can't a backend simply generate code for the large shift, as if it had been a call to Word.Shift(..., 100)? It would be dead/unreachable code, but not a big deal. It looks like the front end skips the code when it can figure it out. I'll have to look into this more, but it is acting different/better than I realized. [Jay] EVAL -FIRST(INTEGER); [Jay] I believe the frontend should issue a warning. [Tony] Why? The front-end does not reason about overflow (except when computing compile-time constants). Overflow is a run-time concept!!!!!!!!!!!!! Because it can often easily prove that overflow will occur. It is worth warning about? if I write: a := 1 + 2; does the front end not optimize that to: a := 3; ? Imho it should 1 + 2 provably at compile time does not overflow. > how to enable overflow > The correct thing to do is introduce different interfaces/modules/types/functions > which either always do overflow checking, or, perhaps but less likely, > new interfaces/modules/types/functions that are runtime configurable, as > INTEGER was originally speced. > NOOOOO!!!!!!! That will impose an undue expense on targets where such checking is expensive. I doubt I suggested what you think I did. If someone really needs overflow checking, then it will cost them whatever it costs them, on whatever target they care about. I am NOT suggesting changing any existing interface/module/type. In fact I am proposing that FloatMode's hypothetical feature to enable overflow checking be removed. That "a + b", "a * b" etc., where a is INTEGER, never ever get overflow checking. If anyone needs it, they'd use some as yet to be specified and implemented type/interface/module. Like INTERFACE IntOv; TYPE T = INTEGER; PROCEDURE Add(a, b: T) RAISES Something: T; etc. or maybe a new compiler-builtin type INTEGEROV. Maybe use the word "safe" as it is popular for this sort of thing (search the web for "safeint"). - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From Highjinks at gmx.com Thu Mar 4 22:29:36 2010 From: Highjinks at gmx.com (Chris) Date: Thu, 4 Mar 2010 22:29:36 +0100 (CET) Subject: [M3devel] Referencing opaque C structs in M3 code? Message-ID: <20100304173345.06cb5d69.Highjinks@gmx.com> Alright, this has thrown me for a bit of a loop...no pun intended... Suppose I'm linking my Modula 3 code with a C library where the public API has a declaration like this... typedef struct opaque_foo_t opaque_foo_t; And the type opaque_foo_t is defined in the private part of the library. Do I need to create a seperate representation of the structure for my Modula3 program or can I just create a pointer for it and pass the pointer around? Assuming of course I dont have to actually reference anything inside opaque_foo_t. Would I just create an UNTRACED REF Ctypes.void_star or something similiar? I'm a little puzzled. Any help would be appreciated. -- Chris From mika at async.async.caltech.edu Thu Mar 4 22:49:17 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Thu, 04 Mar 2010 13:49:17 -0800 Subject: [M3devel] Referencing opaque C structs in M3 code? In-Reply-To: <20100304173345.06cb5d69.Highjinks@gmx.com> References: <20100304173345.06cb5d69.Highjinks@gmx.com> Message-ID: <20100304214917.9D0DD1A2080@async.async.caltech.edu> Ctypes.void_star is already what you want. It's supposed to behave like a (void *), which I think is what you want? You can also use ADDRESS but it's a bit less clear, I think. Mika Chris writes: >Alright, this has thrown me for a bit of a loop...no pun intended... > >Suppose I'm linking my Modula 3 code with a C library where the public API has > a declaration like this... > >typedef struct opaque_foo_t opaque_foo_t; > >And the type opaque_foo_t is defined in the private part of the library. > >Do I need to create a seperate representation of the structure for my Modula3 >program or can I just create a pointer for it and pass the pointer around? Ass >uming of course I dont have to actually reference anything inside opaque_foo_t >. Would I just create an UNTRACED REF Ctypes.void_star or something similiar? > >I'm a little puzzled. Any help would be appreciated. > > > >-- >Chris From hosking at cs.purdue.edu Thu Mar 4 22:59:42 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 4 Mar 2010 16:59:42 -0500 Subject: [M3devel] Referencing opaque C structs in M3 code? In-Reply-To: <20100304173345.06cb5d69.Highjinks@gmx.com> References: <20100304173345.06cb5d69.Highjinks@gmx.com> Message-ID: <7CAE16E7-C170-4A0B-B515-2B2DD5493DBE@cs.purdue.edu> You can assume that ADDRESS is entirely interchangeable with a C void *. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 4 Mar 2010, at 16:29, Chris wrote: > Alright, this has thrown me for a bit of a loop...no pun intended... > > Suppose I'm linking my Modula 3 code with a C library where the public API has a declaration like this... > > typedef struct opaque_foo_t opaque_foo_t; > > And the type opaque_foo_t is defined in the private part of the library. > > Do I need to create a seperate representation of the structure for my Modula3 program or can I just create a pointer for it and pass the pointer around? Assuming of course I dont have to actually reference anything inside opaque_foo_t. Would I just create an UNTRACED REF Ctypes.void_star or something similiar? > > I'm a little puzzled. Any help would be appreciated. > > > > -- > Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 5 02:00:37 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 5 Mar 2010 01:00:37 +0000 Subject: [M3devel] optimizing for size or speed? In-Reply-To: References: , , <20100211120309.GB27402@topoi.pooq.com>, Message-ID: I still find these decisions difficult. Esp. when the option is to call a function or not. Esp. for "builtin" stuff like 64bit math and set operations. - Jay ________________________________ > From: jay.krell at cornell.edu > To: hendrik at topoi.pooq.com; m3devel at elegosoft.com > Date: Thu, 11 Feb 2010 12:41:12 +0000 > Subject: Re: [M3devel] optimizing for size or speed? > > > > > > > > > cache -- good point I had forgotten, thanks. > > > > > > Still: > > use the divide instruction, which is smallest, or multiply by reciprocal (which is generally a multiply and a shift) > > (Given a 32x32=>64 multiply operation. x86 doesn't even have 32x32=>32, only 32x32=>64, I believe.) > > Any 32bit division by a constant can be optimized this way and every C compiler knows it. > > > > > > multiply by a constant using multiply instruction, or decompose into some adds? > > The AMD64 optimization guide suggests speed optimizations where they give a sequence for multiplication for every constant up to 32, some are just to use mul. But many are one or two other instructions. > > > > Multiply by 5 is: > > lea eax,[eax+eax*4] > > > > Multiply by 10 is: > > lea eax,[eax+eax*4] > > add eax,eax > > > > > The AMD64 manual even advises to inline 64bit shifts by a non-constant. > > But I can't get Visual C++ to do that. It always calls a function. > > > > > - Jay > > >> Date: Thu, 11 Feb 2010 07:03:09 -0500 >> From: hendrik at topoi.pooq.com >> To: m3devel at elegosoft.com >> Subject: Re: [M3devel] optimizing for size or speed? >> >> On Thu, Feb 11, 2010 at 08:53:08AM +0000, Jay K wrote: >>> >>> There are all kinds of equivalent code sequences. >>> For the maintainer of m3back to chose among. >> >> In case of doubt, go for size; size all by itself costs time in cache >> misses, paging, etc. >> >> Besides, it's possible to measure space. >> >> -- hendrik From jay.krell at cornell.edu Fri Mar 5 14:59:02 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 5 Mar 2010 13:59:02 +0000 Subject: [M3devel] examples In-Reply-To: <20100227235250.mbm07ehhwsokskg4@mail.elegosoft.com> References: , <20100227235250.mbm07ehhwsokskg4@mail.elegosoft.com> Message-ID: Olaf, it looks like I had it backwards. The release branch is as one would want it. You made the change there. Head lags. Should be ported to head but not pressing. Did the files have to move? Oh well. - Jay > Date: Sat, 27 Feb 2010 23:52:50 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] examples > > Quoting Jay K : > > > > > I think the release branch is missing the change to move examples. > > > > Update it? > Yes, please do. > > Olaf > > (I say this based on differences in the scripts.) > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 5 16:06:00 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 5 Mar 2010 15:06:00 +0000 Subject: [M3devel] release status? Message-ID: release status? - cvsupd hang is uninvestigated - NT386 lacks 64bit longint In head, 64bit longint support is quite good, however I found a bug recently, in head, that needs investigation; This code: C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227\Main.m3(129): result64 := Long.Insert(flipA * a64, flipB * b64, m, n); yields: C:\dev2\cm3.2\m3-sys\m3back\src\M3x86.m3(1045): u.Err("Couldn't find var to free in 'free_temp'"); - Also, head vs. release: TInt/TWord/m3front - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 5 16:36:57 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 5 Mar 2010 15:36:57 +0000 Subject: [M3devel] double double double double checking jmp_buf size/alignment In-Reply-To: <20100106162558.7B4101A2079@async.async.caltech.edu> References: , <20100106162558.7B4101A2079@async.async.caltech.edu> Message-ID: Mika, Hm. I would have expected 44 for FreeBSD/x86. Needs slightly more investigation. PPC_DARWIN we agree on. Thanks, - Jay > To: jay.krell at cornell.edu > Date: Wed, 6 Jan 2010 08:25:58 -0800 > From: mika at async.async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] double double double double checking jmp_buf size/alignment > > Forgot the #endif there? > > (81)trs80:~>./a.out > 48 4 > (82)trs80:~>uname -a > FreeBSD trs80.async.caltech.edu 4.11-RELEASE FreeBSD 4.11-RELEASE #3: Mon Nov 21 20:27:08 PST 2005 root at trs80.async.caltech.edu:/usr/src/sys/compile/TRS80 i386 > > [lapdog:~] mika% cc jay.c > [lapdog:~] mika% ./a.out > 768 4 > [lapdog:~] mika% uname -a > Darwin lapdog 8.11.0 Darwin Kernel Version 8.11.0: Wed Oct 10 18:26:00 PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC Power Macintosh powerpc > [lapdog:~] mika% > > Jay K writes: > >--_b0fc625a-8a60-4f2b-9394-0f52cc975894_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >Was truncated!..edited slightly to avoid.. > > > > > >From: jay.krell at cornell.edu > >To: m3devel at elegosoft.com > >Subject: double double double double checking jmp_buf size/alignment > >Date: Wed=2C 6 Jan 2010 12:28:02 +0000 > > > > > > > > > > > > > > > > > >Getting this right is very important. > > Well=2C we can overstate but it is wasteful. > > > > > >So anyone with any time=2C please compile/run this and send the "platform" = > >(uname -a) and the output=2C thanks. > >Or look at m3-libs/m3core/src/C/*/Csetjmp.i3 and > >m3-sys/m3middle/src/Target.m3 and see if all three agree. > > > > > >I have checked a bunch of systems myself but extra checking is good. > > > > > >If you have a system we do not yet or any longer support=2C those are ok to= > >o. > > (e.g. Alpha_*=2C ARM_*=2C MIPS_*=2C *_Irix=2C *_VMS=2C *_Tru64 etc.) > > > >=20 > >#include > >#include > > > >#ifdef __GNUC__ > >#define ALIGN_OF(x) ((int)(sizeof(struct { char a=3B x b=3B }) - sizeof(x))= > >) > >#else > >#define ALIGN_OF(x) ((int)__alignof(x)) > > > >int main() > >{ > > printf("%d %d\n"=2C (int)sizeof(jmp_buf)=2C ALIGN_OF(jmp_buf))=3B > > return 0=3B > >} > > > > > >Thanks=2C > > - Jay > > = > > > >--_b0fc625a-8a60-4f2b-9394-0f52cc975894_ > >Content-Type: text/html; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > > > > > > > > > >Was truncated!..edited slightly to avoid..


>g">From: jay.krell at cornell.edu
To: m3devel at elegosoft.com
Subject: dou= > >ble double double double checking jmp_buf size/alignment
Date: Wed=2C 6 = > >Jan 2010 12:28:02 +0000

> > > > > > > > > > > > > >Getting this right is very important.
 =3B Well=2C we can overstate = > >but it is wasteful.


So anyone with any time=2C please compile/ru= > >n this and send the "platform" (uname -a) and the output=2C thanks.
Or l= > >ook at m3-libs/m3core/src/C/*/Csetjmp.i3 and
m3-sys/m3middle/src/Target.= > >m3 and see if all three agree.


I have checked a bunch of systems= > > myself but extra checking is good.


If you have a system we do n= > >ot yet or any longer support=2C those are ok too.
 =3B(e.g. Alpha_*= > >=2C ARM_*=2C MIPS_*=2C *_Irix=2C *_VMS=2C *_Tru64 etc.)

 =3B
= > >#include <=3Bsetjmp.h>=3B
#include <=3Bstdio.h>=3B

#ifdef= > > __GNUC__
#define ALIGN_OF(x) ((int)(sizeof(struct { char a=3B x b=3B })= > > - sizeof(x)))
#else
#define ALIGN_OF(x) ((int)__alignof(x))

i= > >nt main()
{
 =3B printf("%d %d\n"=2C (int)sizeof(jmp_buf)=2C ALIG= > >N_OF(jmp_buf))=3B
 =3B return 0=3B
}


Thanks=2C
&nbs= > >p=3B- Jay
> >= > > > >--_b0fc625a-8a60-4f2b-9394-0f52cc975894_-- -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 5 16:40:28 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 5 Mar 2010 15:40:28 +0000 Subject: [M3devel] double double double double checking jmp_buf size/alignment In-Reply-To: References: , Message-ID: Randy, thanks, we agree. Target.m3: | Systems.I386_INTERIX => (* Visual C++'s 16, plus two ints, one to say if sigmask saved, and one to possibly save it. *) Jumpbuf_size := 18 * Address.size; | Systems.NT386, Systems.NT386GNU, Systems.I386_NT, Systems.I386_CYGWIN, Systems.I386_MINGW => (* Cygwin is 13, Visual C++ is 16. Interix is 18. Use 18 for interop. Note that Cygwin's setjmp.h header is wrong, off by a factor of 4. Cygwin provides setjmp and _setjmp that resolve the same. Visual C++ provides only _setjmp. Visual C++ also has _setjmp3 that the compiler generates a call to. In fact _setjmp appears to only use 8 ints and _setjmp3 appears to use more. Consider switching to _setjmp3. *) Jumpbuf_size := (18 * Address.size); - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 6 Jan 2010 15:03:02 -0500 Subject: Re: [M3devel] double double double double checking jmp_buf size/alignment Jay: Results on the following platforms are all identical: Windows 2000, Intel Pentium 3: 64 4 Windows XP, Intel Core Duo T2400: 64 4 Windows Vista, Intel Core2 Duo P9600: 64 4 Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, January 06, 2010 8:18 AM To: m3devel Subject: Re: [M3devel] double double double double checking jmp_buf size/alignment Was truncated!..edited slightly to avoid.. From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: double double double double checking jmp_buf size/alignment Date: Wed, 6 Jan 2010 12:28:02 +0000 Getting this right is very important. Well, we can overstate but it is wasteful. So anyone with any time, please compile/run this and send the "platform" (uname -a) and the output, thanks. Or look at m3-libs/m3core/src/C/*/Csetjmp.i3 and m3-sys/m3middle/src/Target.m3 and see if all three agree. I have checked a bunch of systems myself but extra checking is good. If you have a system we do not yet or any longer support, those are ok too. (e.g. Alpha_*, ARM_*, MIPS_*, *_Irix, *_VMS, *_Tru64 etc.) #include #include #ifdef __GNUC__ #define ALIGN_OF(x) ((int)(sizeof(struct { char a; x b; }) - sizeof(x))) #else #define ALIGN_OF(x) ((int)__alignof(x)) int main() { printf("%d %d\n", (int)sizeof(jmp_buf), ALIGN_OF(jmp_buf)); return 0; } Thanks, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Fri Mar 5 21:38:28 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Fri, 5 Mar 2010 15:38:28 -0500 Subject: [M3devel] optimizing for size or speed? In-Reply-To: References: Message-ID: <20100305203828.GA17524@topoi.pooq.com> On Fri, Mar 05, 2010 at 01:00:37AM +0000, Jay K wrote: > > I still find these decisions difficult. > Esp. when the option is to call a function or not. > Esp. for "builtin" stuff like 64bit math and set operations. > > - Jay That's right. They can be difficult. -- hendrik From hendrik at topoi.pooq.com Fri Mar 5 22:03:10 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Fri, 5 Mar 2010 16:03:10 -0500 Subject: [M3devel] Referencing opaque C structs in M3 code? In-Reply-To: <7CAE16E7-C170-4A0B-B515-2B2DD5493DBE@cs.purdue.edu> References: <20100304173345.06cb5d69.Highjinks@gmx.com> <7CAE16E7-C170-4A0B-B515-2B2DD5493DBE@cs.purdue.edu> Message-ID: <20100305210310.GB17524@topoi.pooq.com> On Thu, Mar 04, 2010 at 04:59:42PM -0500, Tony Hosking wrote: > You can assume that ADDRESS is entirely interchangeable with a C void *. What struct opaque_foo_t provides that void doesn't is some further type-checking -- struct opaque_foo_t is a different type from struct other_foo_t. Presumably you should use Modula 3 techniques for making an opaque types in your interface. - hendrik > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 4 Mar 2010, at 16:29, Chris wrote: > > > Alright, this has thrown me for a bit of a loop...no pun intended... > > > > Suppose I'm linking my Modula 3 code with a C library where the public API has a declaration like this... > > > > typedef struct opaque_foo_t opaque_foo_t; > > > > And the type opaque_foo_t is defined in the private part of the library. > > > > Do I need to create a seperate representation of the structure for my Modula3 program or can I just create a pointer for it and pass the pointer around? Assuming of course I dont have to actually reference anything inside opaque_foo_t. Would I just create an UNTRACED REF Ctypes.void_star or something similiar? > > > > I'm a little puzzled. Any help would be appreciated. > > > > > > > > -- > > Chris > From hendrik at topoi.pooq.com Sat Mar 6 07:22:08 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Sat, 6 Mar 2010 01:22:08 -0500 Subject: [M3devel] RC4 Message-ID: <20100306062208.GA16974@topoi.pooq.com> I installed a few packages of RC4 a while ago for Debiaan Squeeze Linux on intel 32-bit, and have had no problems with them. They work just fine, I installed the core package, and then gui and m3gdb. WHile installing I noticed the following: The installation instructions told me to use a new empty directory to unpack packages. Is it a new empty directory for each package, or can I use the same one for all packages? If I use the same one, should I empty it before each package? And should this be different from the directory where I unpacked cm3-bin-ws-m3devtool-TARGET-VERSION.tgz? I made a new directory to unpack each package, but the instructions should be more explicit. There should be some mention which packages depend on which others. I suspect, for example, that juno requires gui. Such constraints should be mentioned in the installatioin instructions. -- hendrik From hendrik at topoi.pooq.com Sat Mar 6 07:28:06 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Sat, 6 Mar 2010 01:28:06 -0500 Subject: [M3devel] FW: optimizing range checks? In-Reply-To: References: <20100302125931.619312474001@birch.elegosoft.com> Message-ID: <20100306062806.GB16974@topoi.pooq.com> On Tue, Mar 02, 2010 at 01:16:03PM +0000, Jay K wrote: > > Hey that's pretty clever. > > It costs a register, but given: > > > > if (b >= constantX|| b <= -constantY) > a = 0; > > > > The C compiler instead does something like: > > if ((b + constantY - 1) > (constantX + constantY - 1)) > > a = 0; > > > > This is something the front end could do in many cases.? > > Adding a constant to eliminate one of the compares and branches is a win. > > If an x86 compiler will give up a register for this, then it is probably a win everywhere. > > > > Granted, it probably requires silent overflow. Oh well. It requires unsigned arithmetic; in particular the final compare has to be unsigned. -- hendrik From hosking at cs.purdue.edu Sat Mar 6 16:12:47 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sat, 6 Mar 2010 10:12:47 -0500 Subject: [M3devel] Referencing opaque C structs in M3 code? In-Reply-To: <20100305210310.GB17524@topoi.pooq.com> References: <20100304173345.06cb5d69.Highjinks@gmx.com> <7CAE16E7-C170-4A0B-B515-2B2DD5493DBE@cs.purdue.edu> <20100305210310.GB17524@topoi.pooq.com> Message-ID: <20C86CDF-49BA-4CA8-918A-D048F110BB7F@cs.purdue.edu> You can have a TYPE foo = BRANDED "foo" ADDRESS to distinguish from other void types. On 5 Mar 2010, at 16:03, hendrik at topoi.pooq.com wrote: > On Thu, Mar 04, 2010 at 04:59:42PM -0500, Tony Hosking wrote: >> You can assume that ADDRESS is entirely interchangeable with a C void *. > > What struct opaque_foo_t provides that void doesn't is some further > type-checking -- struct opaque_foo_t is a different type from struct > other_foo_t. > > Presumably you should use Modula 3 techniques for making an opaque > types in your interface. > > - hendrik > >> >> Antony Hosking | Associate Professor | Computer Science | Purdue University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> On 4 Mar 2010, at 16:29, Chris wrote: >> >>> Alright, this has thrown me for a bit of a loop...no pun intended... >>> >>> Suppose I'm linking my Modula 3 code with a C library where the public API has a declaration like this... >>> >>> typedef struct opaque_foo_t opaque_foo_t; >>> >>> And the type opaque_foo_t is defined in the private part of the library. >>> >>> Do I need to create a seperate representation of the structure for my Modula3 program or can I just create a pointer for it and pass the pointer around? Assuming of course I dont have to actually reference anything inside opaque_foo_t. Would I just create an UNTRACED REF Ctypes.void_star or something similiar? >>> >>> I'm a little puzzled. Any help would be appreciated. >>> >>> >>> >>> -- >>> Chris >> From jay.krell at cornell.edu Sun Mar 7 10:12:38 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 7 Mar 2010 09:12:38 +0000 Subject: [M3devel] word.insert/extract warnings/optimizations Message-ID: IO.PutInt(Word.Insert(a,b,20,20)); IO.PutInt(Word.Extract(1,1,50)); IO.PutInt(Word.Extract(1,30,30)); Tony, there's discrepancy between Word.Extract and Word.Insert. m3front does a better job of checking and optimizing Word.Extract. For Insert it always does a runtime add, and a runtime range check against the number of bits in integer. It isn't wrong, just could be better. For Extract, it checks either constant against the number of bits, if they are both constant, it checks the sum and calls extract_mn. If they are both constant and the sum is too large, it just generates code to cause a range fault at runtime check_hi(1 vs. 0). Probably that could be a little more direct. If one is constant it calls extract_n, or a slightly optimized use of extract if the other is constant. I'll probably tackle this soon. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 7 12:25:10 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 7 Mar 2010 11:25:10 +0000 Subject: [M3devel] -no-m3ship-resolution Message-ID: This is head. Two problems here. One the feature apparently not fully working. I should look at that. Two the test is too platform specific -- lib.lib vs. liblib.a. Maybe reasonable to support some text substitution over the results to address that. Or heck, maybe abstract out the "naming conventions"? Maybe too difficult and too little gain. I'm surprised even have TARGET like that. The main point I think is the install root. --- ../src/p2/p221/stdout.build 2009-12-15 03:04:01.000000000 -0800 +++ ../src/p2/p221/NT386/stdout.build 2010-03-07 03:11:39.531250000 -0800 @@ -1,7 +1,7 @@ -make_dir(PKG_INSTALL & "/p221/" & TARGET) -install_file(".M3EXPORTS", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file("liblib.a", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file("liblib.m3x", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file(".M3WEB", PKG_INSTALL & "/p221/" & TARGET, "0644") -make_dir(PKG_INSTALL & "/p221") -install_file("../Main.m3", PKG_INSTALL & "/p221", "0644") +make_dir("/cm3/pkg/p221/" & TARGET) +install_file(".M3EXPORTS", "/cm3/pkg/p221/" & TARGET, "0644") +install_file("lib.lib", "/cm3/pkg/p221/" & TARGET, "0644") +install_file("lib.m3x", "/cm3/pkg/p221/" & TARGET, "0644") +install_file(".M3WEB", "/cm3/pkg/p221/" & TARGET, "0644") +make_dir("/cm3/pkg/p221") +install_file("../Main.m3", "/cm3/pkg/p221", "0644") --- p222 --- .M3SHIP Library - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 7 12:48:59 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 7 Mar 2010 11:48:59 +0000 Subject: [M3devel] -no-m3ship-resolution In-Reply-To: References: Message-ID: My CM3_INSTALL environment variable has forward slashes. It'd probably work otherwise. One fix is: Index: M3Build.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/cm3/src/M3Build.m3,v retrieving revision 1.31 diff -u -r1.31 M3Build.m3 --- M3Build.m3 4 Sep 2009 10:25:26 -0000 1.31 +++ M3Build.m3 7 Mar 2010 11:40:50 -0000 @@ -133,7 +133,7 @@ (* some more config dependent backward compatibility hacks... *) Quake.Define (t, "M3SEARCH_TABLES", "-T" & M3TFile); Quake.Define (t, "DEFAULT_BUILD_DIR", GetConfig (t, "BUILD_DIR")); - Quake.Define (t, "M3", M3Path.New (GetConfig (t, "BIN_USE"), "cm3")); + Quake.Define (t, "M3", TextUtils.Substitute(M3Path.New (GetConfig (t, "BIN_USE"), "cm3"), "/", M3Path.SlashText)); Quake.Define (t, "PACKAGE_DIR", pkg); t.build_pkg := M3ID.Add (Pathname.Last (pkg)); @@ -143,19 +143,19 @@ (* M3Path.New is used to canonicalize the paths -- to remove dots *) - t.pkg_use := M3Path.New (GetConfig (t, "PKG_USE")); + t.pkg_use := TextUtils.Substitute(M3Path.New (GetConfig (t, "PKG_USE")), "/", M3Path.SlashText); (* not in Quake.Machine - t.bin_use := M3Path.New (GetConfig (t, "BIN_USE")); - t.lib_use := M3Path.New (GetConfig (t, "LIB_USE")); + t.bin_use := TextUtils.Substitute(M3Path.New (GetConfig (t, "BIN_USE")), "/", M3Path.SlashText); + t.lib_use := TextUtils.Substitute(M3Path.New (GetConfig (t, "LIB_USE")), "/", M3Path.SlashText); *) - t.pkg_install := M3Path.New (GetConfig (t, "PKG_INSTALL")); - t.install_root := M3Path.New (GetConfig (t, "INSTALL_ROOT")); - t.bin_install := M3Path.New (GetConfig (t, "BIN_INSTALL")); - t.lib_install := M3Path.New (GetConfig (t, "LIB_INSTALL")); - t.emacs_install := M3Path.New (GetConfig (t, "EMACS_INSTALL")); - t.doc_install := M3Path.New (GetConfig (t, "DOC_INSTALL")); - t.man_install := M3Path.New (GetConfig (t, "MAN_INSTALL")); - t.html_install := M3Path.New (GetConfig (t, "HTML_INSTALL")); + t.pkg_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "PKG_INSTALL")), "/", M3Path.SlashText); + t.install_root := TextUtils.Substitute(M3Path.New (GetConfig (t, "INSTALL_ROOT")), "/", M3Path.SlashText); + t.bin_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "BIN_INSTALL")), "/", M3Path.SlashText); + t.lib_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "LIB_INSTALL")), "/", M3Path.SlashText); + t.emacs_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "EMACS_INSTALL")), "/", M3Path.SlashText); + t.doc_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "DOC_INSTALL")), "/", M3Path.SlashText); + t.man_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "MAN_INSTALL")), "/", M3Path.SlashText); + t.html_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "HTML_INSTALL")), "/", M3Path.SlashText); t.have_pkgtools := GetConfigBool (t, "HAVE_PKGTOOLS"); t.at_SRC := GetConfigBool (t, "AT_SRC"); t.system_liborder := QVal.ToArray (t, ConfigDefn (t, "SYSTEM_LIBORDER").value); but I'm not sure I like that. I think probably we should have parallel variables bin_install_forwardslash, etc., in which \\ is replaced by /. If we compute the paths ourselves on Win32, we use \\. But if user sets them, we leave them alone. Either slash works. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Sun, 7 Mar 2010 11:25:10 +0000 Subject: [M3devel] -no-m3ship-resolution This is head. Two problems here. One the feature apparently not fully working. I should look at that. Two the test is too platform specific -- lib.lib vs. liblib.a. Maybe reasonable to support some text substitution over the results to address that. Or heck, maybe abstract out the "naming conventions"? Maybe too difficult and too little gain. I'm surprised even have TARGET like that. The main point I think is the install root. --- ../src/p2/p221/stdout.build 2009-12-15 03:04:01.000000000 -0800 +++ ../src/p2/p221/NT386/stdout.build 2010-03-07 03:11:39.531250000 -0800 @@ -1,7 +1,7 @@ -make_dir(PKG_INSTALL & "/p221/" & TARGET) -install_file(".M3EXPORTS", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file("liblib.a", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file("liblib.m3x", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file(".M3WEB", PKG_INSTALL & "/p221/" & TARGET, "0644") -make_dir(PKG_INSTALL & "/p221") -install_file("../Main.m3", PKG_INSTALL & "/p221", "0644") +make_dir("/cm3/pkg/p221/" & TARGET) +install_file(".M3EXPORTS", "/cm3/pkg/p221/" & TARGET, "0644") +install_file("lib.lib", "/cm3/pkg/p221/" & TARGET, "0644") +install_file("lib.m3x", "/cm3/pkg/p221/" & TARGET, "0644") +install_file(".M3WEB", "/cm3/pkg/p221/" & TARGET, "0644") +make_dir("/cm3/pkg/p221") +install_file("../Main.m3", "/cm3/pkg/p221", "0644") --- p222 --- .M3SHIP Library - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 7 14:37:28 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 7 Mar 2010 13:37:28 +0000 Subject: [M3devel] test p078 works in release but not head -- NUMBER(open array constant) not constant Message-ID: C:\dev2\cm3.2\m3-sys\m3tests\src\p0\p078 This compiles with release but not head. (* Copyright (C) 1994, Digital Equipment Corporation. *) (* All rights reserved. *) (* See the file COPYRIGHT for a full description. *) MODULE Main; FROM Test IMPORT done, checkI, checkB; CONST Numbers = ARRAY OF INTEGER {2, 3, 5, 7, 11}; First = Numbers [FIRST (Numbers)]; Last = Numbers [LAST (Numbers)]; Number = NUMBER (Numbers); Empty = ARRAY OF INTEGER {}; EFirst = FIRST (Empty); ELast = LAST (Empty); ENumber = NUMBER (Empty); BEGIN checkI (First, 2); checkI (Last, 11); checkI (Number, 5); checkB (EFirst > ELast, TRUE); checkI (ENumber, 0); done (); END Main. new source -> compiling Main.m3 "..\Main.m3", line 14: value is not constant "..\Main.m3", line 19: value is not constant 2 errors encountered compilation failed => not building program "pgm.exe" Fatal Error: package build failed C:\dev2\cm3.2\m3-sys\m3tests\src\p0\p078> It is the lines with "NUMBER". - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 7 15:14:29 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 7 Mar 2010 14:14:29 +0000 Subject: [M3devel] test p078 works in release but not head -- NUMBER(open array constant) not constant In-Reply-To: References: Message-ID: nevermind. Index: Number.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/builtinOps/Number.m3,v retrieving revision 1.5 diff -u -r1.5 Number.m3 --- Number.m3 18 Feb 2010 02:33:09 -0000 1.5 +++ Number.m3 7 Mar 2010 14:11:17 -0000 @@ -102,8 +102,8 @@ IF ArrayExpr.GetBounds (e, min, max) AND TInt.Subtract (max, min, tmp) AND TInt.Add (tmp, TInt.One, num) - AND NOT TInt.LT (num, Target.Integer.max) - AND NOT TInt.LT (Target.Integer.max, num) + AND TInt.GE (num, Target.Integer.min) + AND TInt.LE (num, Target.Integer.max) THEN RETURN IntegerExpr.New (Int.T, num); ELSE RETURN NIL; END; Notice how the full assortment of LE/GE/LT/GT/EQ/NE makes it easier to read and write the code and get it correct. :) Of course, the "double max" stands out as incorrect either way. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu; m3devel at elegosoft.com Date: Sun, 7 Mar 2010 13:37:28 +0000 Subject: [M3devel] test p078 works in release but not head -- NUMBER(open array constant) not constant C:\dev2\cm3.2\m3-sys\m3tests\src\p0\p078 This compiles with release but not head. (* Copyright (C) 1994, Digital Equipment Corporation. *) (* All rights reserved. *) (* See the file COPYRIGHT for a full description. *) MODULE Main; FROM Test IMPORT done, checkI, checkB; CONST Numbers = ARRAY OF INTEGER {2, 3, 5, 7, 11}; First = Numbers [FIRST (Numbers)]; Last = Numbers [LAST (Numbers)]; Number = NUMBER (Numbers); Empty = ARRAY OF INTEGER {}; EFirst = FIRST (Empty); ELast = LAST (Empty); ENumber = NUMBER (Empty); BEGIN checkI (First, 2); checkI (Last, 11); checkI (Number, 5); checkB (EFirst > ELast, TRUE); checkI (ENumber, 0); done (); END Main. new source -> compiling Main.m3 "..\Main.m3", line 14: value is not constant "..\Main.m3", line 19: value is not constant 2 errors encountered compilation failed => not building program "pgm.exe" Fatal Error: package build failed C:\dev2\cm3.2\m3-sys\m3tests\src\p0\p078> It is the lines with "NUMBER". - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Mon Mar 8 10:05:47 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 08 Mar 2010 10:05:47 +0100 Subject: [M3devel] RC4 In-Reply-To: <20100306062208.GA16974@topoi.pooq.com> References: <20100306062208.GA16974@topoi.pooq.com> Message-ID: <20100308100547.2ytic2gw8wcwwo0k@mail.elegosoft.com> Quoting hendrik at topoi.pooq.com: > I installed a few packages of RC4 a while ago for Debiaan Squeeze Linux > on intel 32-bit, and have had no problems with them. They work just > fine, > > I installed the core package, and then gui and m3gdb. Fine. > WHile installing I noticed the following: > > The installation instructions told me to use a new empty > directory to unpack packages. Is it a new empty directory for each > package, or can I use the same one for all packages? If I use the > same one, should I empty it before each package? And should this be > different from the directory where I unpacked > cm3-bin-ws-m3devtool-TARGET-VERSION.tgz? I made a new directory to > unpack each package, but the instructions should be more explicit. I'll try to improve that for RC5. > There should be some mention which packages depend on which others. I > suspect, for example, that juno requires gui. Such constraints should > be mentioned in the installatioin instructions. You mean something like http://www.modula3.com/cm3/releng/collection-juno.html ? Probably the link to the package descriptions is too hidden in the table (last column). Thanks for the feedback, Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Mon Mar 8 10:15:59 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 8 Mar 2010 09:15:59 +0000 Subject: [M3devel] m3cg.i3 reduction? Message-ID: The backend interface has a few aspects that the frontend does not use. Implementations of these are therefore extremely difficult to test and therefore probably don't work. I propose: extract and extract_n should not take a sign_extend:BOOLEAN parameter. It is always false. extract_mn shall retain its sign_extend:BOOLEAN parameter. Though really, it could go too. The front end could just issue divide and the backends could probably figure it out. I like the frontend doing the work though. (Really, if it going to optimize divide by a power of 2, it might be nice to compute reciprocals too...on my list for m3back..) The integer parameters to extract*/insert* should be CARDINAL instead of INTEGER. Or [0..63], with the "63" abstracted out somehow and comments that clarify it is really sometimes only 0..31. The front end already issues range checks for these parameters. The backend shouldn't worry about such wide ranges. Arguably remove insert_m/n and extract_m/n and just have insert/extract. The NT386 backend already notices when parameters on the stack are constants, and the gcc backend probably does too, at least when optimizing. The "specializations" do make it easier for a hypothetical backend simpler than the NT386 to optimize a bit. So I'm on the fence there. zero_n should be removed. It isn't used. Far more radically, I'm suspicious that the use of a stack is a good idea. It'd probably be just as easy or easier, and lead to better code, to have a *sort* of union that encapsulates constants and variables, and pass each "operation" (add/multiply/subtract/insert/extract/etc.) its parameters with the function all. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 8 15:59:41 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 8 Mar 2010 09:59:41 -0500 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: References: Message-ID: <28E061D6-8DB0-4C69-A23D-F460573E16E0@cs.purdue.edu> On 8 Mar 2010, at 04:15, Jay K wrote: > The backend interface has a few aspects that the frontend does not use. > Implementations of these are therefore extremely difficult to test and therefore probably don't work. This is inevitable. Your backend is not implementing an interface that the front-end defines. It is implementing an interface defined in m3middle which is much wider than that used by m3front. For good reason, we should not dumb down m3middle just because m3front doesn't make use of all its operations. Similarly, you should not assume that m3front will not make use of some operations in future. This is good practice for separation of concerns in support of modularity. > I propose: > > > extract and extract_n should not take a sign_extend:BOOLEAN parameter. > It is always false. Only as currently generated by m3front. > extract_mn shall retain its sign_extend:BOOLEAN parameter. For consistency and generality we should retain it for all extract operators. > Though really, it could go too. > The front end could just issue divide and the backends could probably > figure it out. I like the frontend doing the work though. The front-end should not be concerned with optimisation. It's job is semantics, and having extract/insert is important for some targets. > (Really, if it going to optimize divide by a power of 2, it might be nice > to compute reciprocals too...on my list for m3back..) Same. Front-ends should not be worried about optimising. > The integer parameters to extract*/insert* should be CARDINAL instead of INTEGER. > Or [0..63], with the "63" abstracted out somehow and comments that clarify it is > really sometimes only 0..31. That is not a general interface. Just because m3front filters the operands it provides doesn't mean that all upstream clients of m3middle will do so. > The front end already issues range checks for these parameters. > The backend shouldn't worry about such wide ranges. The backend should be prepared for them. > Arguably remove insert_m/n and extract_m/n and just have insert/extract. That is a possibility. But again, some backends might find it convenient to know about constant operands. Let's not constrain things. > The NT386 backend already notices when parameters on the stack are constants, > and the gcc backend probably does too, at least when optimizing. > The "specializations" do make it easier for a hypothetical backend simpler > than the NT386 to optimize a bit. Precisely! > So I'm on the fence there. We should retain them. > Zero_n should be removed. I've considered using it for some initialisation support. We don't want to throw something out that may later be useful. M3CG was designed for generality, and actually even predates the Modula-3 language itself. It is derived from code generators at DEC SRC from the 1980s. > It isn't used. Yet... > Far more radically, I'm suspicious that the use of a stack is a good idea. > It'd probably be just as easy or easier, and lead to better code, to > have a *sort* of union that encapsulates constants and variables, > and pass each "operation" (add/multiply/subtract/insert/extract/etc.) its > parameters with the function all. Stack models versus "registers" have always been argued about in compiling. Witness the Java VM spec's use of a stack as compared to the fact that most Java VMs convert the stack to registers for specific targets. > > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Mar 8 17:17:24 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 8 Mar 2010 16:17:24 +0000 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: <28E061D6-8DB0-4C69-A23D-F460573E16E0@cs.purdue.edu> References: , <28E061D6-8DB0-4C69-A23D-F460573E16E0@cs.purdue.edu> Message-ID: > Front-ends should not be worried about optimising. But it does so much already. It unrolls comparisons of "solid" types to a certain extent. It took me a while to track that down. It converts divides by powers of two into shift/extract. m3front (or m3middle?) could act as "shared code" for multiple backends. Computing the reciprocal is something that would be common to various backends. The computation itself is portable. The backend could chose to use it or not (maybe some processor has a fast divide instruction..or equally slow divide and mutiply...). How about TInt.Reciprocal or somesuch? > doesn't mean that all upstream clients of m3middle will do so What other clients of m3middle could possibly exist? And that really need more functionality than m3front needs? What does insert/extract with negative numbers mean? I'll see if the .i3 files describes it in comments. Please let's not just invent general generalities, that we don't use, and can't test. What m3front needs, we should do. What m3front doesn't need, we should not do. m3middle serves m3front. If/when m3front needs it, do it then. Generalities produce dead untestable buggy code and wasted time. There was definitely some in m3back, e.g. sign extended extract seemed wrong for count = 0. I forgot another suggestion: set_eq and set_ne are implemented the same by the two backends, by calling memcmp. All the other comparisons are the same too, via calling functions. Maybe the frontend should just know about this and call the functions? These are unusually high level operations in the backends, that neither one implements in an at all clever way. - Jay From: hosking at cs.purdue.edu Date: Mon, 8 Mar 2010 09:59:41 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] m3cg.i3 reduction? On 8 Mar 2010, at 04:15, Jay K wrote: The backend interface has a few aspects that the frontend does not use. Implementations of these are therefore extremely difficult to test and therefore probably don't work. This is inevitable. Your backend is not implementing an interface that the front-end defines. It is implementing an interface defined in m3middle which is much wider than that used by m3front. For good reason, we should not dumb down m3middle just because m3front doesn't make use of all its operations. Similarly, you should not assume that m3front will not make use of some operations in future. This is good practice for separation of concerns in support of modularity. I propose: extract and extract_n should not take a sign_extend:BOOLEAN parameter. It is always false. Only as currently generated by m3front. extract_mn shall retain its sign_extend:BOOLEAN parameter. For consistency and generality we should retain it for all extract operators. Though really, it could go too. The front end could just issue divide and the backends could probably figure it out. I like the frontend doing the work though. The front-end should not be concerned with optimisation. It's job is semantics, and having extract/insert is important for some targets. (Really, if it going to optimize divide by a power of 2, it might be nice to compute reciprocals too...on my list for m3back..) Same. Front-ends should not be worried about optimising. The integer parameters to extract*/insert* should be CARDINAL instead of INTEGER. Or [0..63], with the "63" abstracted out somehow and comments that clarify it is really sometimes only 0..31. That is not a general interface. Just because m3front filters the operands it provides doesn't mean that all upstream clients of m3middle will do so. The front end already issues range checks for these parameters. The backend shouldn't worry about such wide ranges. The backend should be prepared for them. Arguably remove insert_m/n and extract_m/n and just have insert/extract. That is a possibility. But again, some backends might find it convenient to know about constant operands. Let's not constrain things. The NT386 backend already notices when parameters on the stack are constants, and the gcc backend probably does too, at least when optimizing. The "specializations" do make it easier for a hypothetical backend simpler than the NT386 to optimize a bit. Precisely! So I'm on the fence there. We should retain them. Zero_n should be removed. I've considered using it for some initialisation support. We don't want to throw something out that may later be useful. M3CG was designed for generality, and actually even predates the Modula-3 language itself. It is derived from code generators at DEC SRC from the 1980s. It isn't used. Yet... Far more radically, I'm suspicious that the use of a stack is a good idea. It'd probably be just as easy or easier, and lead to better code, to have a *sort* of union that encapsulates constants and variables, and pass each "operation" (add/multiply/subtract/insert/extract/etc.) its parameters with the function all. Stack models versus "registers" have always been argued about in compiling. Witness the Java VM spec's use of a stack as compared to the fact that most Java VMs convert the stack to registers for specific targets. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Mon Mar 8 13:25:36 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Mon, 8 Mar 2010 07:25:36 -0500 Subject: [M3devel] RC4 In-Reply-To: <20100308100547.2ytic2gw8wcwwo0k@mail.elegosoft.com> References: <20100306062208.GA16974@topoi.pooq.com> <20100308100547.2ytic2gw8wcwwo0k@mail.elegosoft.com> Message-ID: <20100308122536.GA6668@topoi.pooq.com> On Mon, Mar 08, 2010 at 10:05:47AM +0100, Olaf Wagner wrote: > Quoting hendrik at topoi.pooq.com: > > >I installed a few packages of RC4 a while ago for Debiaan Squeeze Linux > >on intel 32-bit, and have had no problems with them. They work just > >fine, > > > >I installed the core package, and then gui and m3gdb. > > Fine. > > >WHile installing I noticed the following: > > > >The installation instructions told me to use a new empty > >directory to unpack packages. Is it a new empty directory for each > >package, or can I use the same one for all packages? If I use the > >same one, should I empty it before each package? And should this be > >different from the directory where I unpacked > >cm3-bin-ws-m3devtool-TARGET-VERSION.tgz? I made a new directory to > >unpack each package, but the instructions should be more explicit. > > I'll try to improve that for RC5. > > >There should be some mention which packages depend on which others. I > >suspect, for example, that juno requires gui. Such constraints should > >be mentioned in the installatioin instructions. > > You mean something like > > http://www.modula3.com/cm3/releng/collection-juno.html Yes. Something like that! I've never seen that page before. > > ? Probably the link to the package descriptions is too hidden in the table > (last column). What table? > > Thanks for the feedback, > > Olaf It looks like a website navigability problem. As installer, I see the archive descriptions in http://www.modula3.com/cm3/releng/ where the Juno line says, cm3-bin-ws-juno-*.tgz source/binary, application, optional A constraint-based graphical editor and there isn't any mention of a package description. -- hendrik > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > From hendrik at topoi.pooq.com Mon Mar 8 13:37:46 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Mon, 8 Mar 2010 07:37:46 -0500 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: <28E061D6-8DB0-4C69-A23D-F460573E16E0@cs.purdue.edu> References: <28E061D6-8DB0-4C69-A23D-F460573E16E0@cs.purdue.edu> Message-ID: <20100308123746.GB6668@topoi.pooq.com> On Mon, Mar 08, 2010 at 09:59:41AM -0500, Tony Hosking wrote: > > M3CG was designed for generality, and actually even predates the > Modula-3 language itself. It is derived from code generators at DEC > SRC from the 1980s. I've been thinking for a while that the Modula 3 back end and a fair bit of its run-time system are probably good enough to be used by other garbage-collected languages. Almost no one else does a good job of combining parallel processing with garbage collection, for example. -- hendrik From rodney_bates at lcwb.coop Mon Mar 8 18:42:16 2010 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 08 Mar 2010 11:42:16 -0600 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: References: Message-ID: <4B9536F8.7000104@lcwb.coop> It is never realistic to try to thoroughly test a later (than the first) phase of any compiler solely by pumping source code through the earlier phases. Every compiler I have ever worked on had tools to allow hand-coded input to later phases, either already, or else I wrote one, if I had significant responsibility for testing. Sometimes it can be done with a plain old text editor, if the intermediate stream is character-encoded. Similarly, you need to be able dump the intermediate forms of output from earlier phases in human-readable form. Don't we already have at least some of this? Jay K wrote: > The backend interface has a few aspects that the frontend does not use. > Implementations of these are therefore /extremely/ difficult to test and > therefore probably don't work. > > > From hosking at cs.purdue.edu Mon Mar 8 20:32:53 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 8 Mar 2010 14:32:53 -0500 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: <4B9536F8.7000104@lcwb.coop> References: <4B9536F8.7000104@lcwb.coop> Message-ID: On 8 Mar 2010, at 12:42, Rodney M. Bates wrote: > It is never realistic to try to thoroughly test a later (than the first) phase of > any compiler solely by pumping source code through the earlier phases. Every > compiler I have ever worked on had tools to allow hand-coded input to later > phases, either already, or else I wrote one, if I had significant responsibility > for testing. Sometimes it can be done with a plain old text editor, if the > intermediate stream is character-encoded. Indeed. > Similarly, you need to be able dump the intermediate forms of output from earlier > phases in human-readable form. Don't we already have at least some of this? Yes, m3cgcat and friends... > > Jay K wrote: >> The backend interface has a few aspects that the frontend does not use. >> Implementations of these are therefore /extremely/ difficult to test and therefore probably don't work. >> From jay.krell at cornell.edu Tue Mar 9 03:03:27 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 02:03:27 +0000 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: References: , <4B9536F8.7000104@lcwb.coop>, Message-ID: There is no such mode currently for m3back. I think. At least it doesn't operate that way normally. It keeps very little in memory, going fairly directly from function calls to file output. Not entirely, but largely. Not a bad idea though. But maybe I'm wrong here. Maybe the thing to write/read to the gcc backend can also drive m3back? And then I can start with m3front output but edit in variations? That could be handy. Still, these things are give and take. You theorize as to what you might need. Then you write the consumer. Then you discover you might have missed some things and might have put in some unnecessary things. Not all generalizations should be kept. Interfaces should be reviewed for areas that ended up not needed. And you also write the producer and find some things to be easy or difficult. "Build it and they will come", to some extent yes, to some extent no. sign extension was "so easy" that the original authors wrote it, including for non-constants, but I don't think it works for count=0 for example, which the current front end never exercises..not even with constants.. Given my current implementation strategy, which produces less efficient but clear and dependency-free code, it is perhaps easier to extend things back. (still waiting to hear if the larger size bothers anyone, though the tables are gone, that is a reduction; not sure clarity is so important at the assembly level, it is assembly after all) - Jay ---------------------------------------- > From: hosking at cs.purdue.edu > Date: Mon, 8 Mar 2010 14:32:53 -0500 > To: rodney_bates at lcwb.coop > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] m3cg.i3 reduction? > > On 8 Mar 2010, at 12:42, Rodney M. Bates wrote: > >> It is never realistic to try to thoroughly test a later (than the first) phase of >> any compiler solely by pumping source code through the earlier phases. Every >> compiler I have ever worked on had tools to allow hand-coded input to later >> phases, either already, or else I wrote one, if I had significant responsibility >> for testing. Sometimes it can be done with a plain old text editor, if the >> intermediate stream is character-encoded. > > Indeed. > >> Similarly, you need to be able dump the intermediate forms of output from earlier >> phases in human-readable form. Don't we already have at least some of this? > > Yes, m3cgcat and friends... > >> >> Jay K wrote: >>> The backend interface has a few aspects that the frontend does not use. >>> Implementations of these are therefore /extremely/ difficult to test and therefore probably don't work. >>> > From jay.krell at cornell.edu Tue Mar 9 06:09:19 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 05:09:19 +0000 Subject: [M3devel] m3_build vs. build in parse.c? In-Reply-To: <16ADF5CA-B729-40CC-ADAB-5FA170C9084B@cs.purdue.edu> References: , <16ADF5CA-B729-40CC-ADAB-5FA170C9084B@cs.purdue.edu> Message-ID: I understand why use m3_build instead of build. But why ever use build? Just when m3_build has no chance of optimizing? Or an oversight? - Jay ________________________________ > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > CC: m3commit at elegosoft.com > Subject: RE: [M3commit] m3_build vs. build in parse.c? > Date: Thu, 4 Mar 2010 08:46:06 +0000 > > > Why not always call m3_build? > > Why ever call build? > > > > - Jay > >> From: hosking at cs.purdue.edu >> Date: Thu, 4 Mar 2010 02:58:30 -0500 >> To: jay.krell at cornell.edu >> CC: m3commit at elegosoft.com >> Subject: Re: [M3commit] m3_build vs. build in parse.c? >> >> So that we can avoid having to analyse all the constants ourselves. We can simply generate the trees and then have gcc fold them. >> >> On 4 Mar 2010, at 02:16, Jay K wrote: >> >>> Why use one vs. the other? >>> It appears that they are equivalent *except* that m3_build attempts to optimize, >>> but falls back to build if it can't. >>> >>> That is, m3_build calls fold_build. >>> >>> - Jay >>> >>> >>> >>> >> From hosking at cs.purdue.edu Tue Mar 9 06:33:07 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 9 Mar 2010 00:33:07 -0500 Subject: [M3devel] m3_build vs. build in parse.c? In-Reply-To: References: , <16ADF5CA-B729-40CC-ADAB-5FA170C9084B@cs.purdue.edu> Message-ID: <556CE042-9DBD-4432-BBD8-82E0A2DD5801@cs.purdue.edu> Not sure if it break anything or not, but no good reason I know of. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 9 Mar 2010, at 00:09, Jay K wrote: > > I understand why use m3_build instead of build. > But why ever use build? Just when m3_build has > no chance of optimizing? Or an oversight? > > - Jay > > ________________________________ >> From: jay.krell at cornell.edu >> To: hosking at cs.purdue.edu >> CC: m3commit at elegosoft.com >> Subject: RE: [M3commit] m3_build vs. build in parse.c? >> Date: Thu, 4 Mar 2010 08:46:06 +0000 >> >> >> Why not always call m3_build? >> >> Why ever call build? >> >> >> >> - Jay >> >>> From: hosking at cs.purdue.edu >>> Date: Thu, 4 Mar 2010 02:58:30 -0500 >>> To: jay.krell at cornell.edu >>> CC: m3commit at elegosoft.com >>> Subject: Re: [M3commit] m3_build vs. build in parse.c? >>> >>> So that we can avoid having to analyse all the constants ourselves. We can simply generate the trees and then have gcc fold them. >>> >>> On 4 Mar 2010, at 02:16, Jay K wrote: >>> >>>> Why use one vs. the other? >>>> It appears that they are equivalent *except* that m3_build attempts to optimize, >>>> but falls back to build if it can't. >>>> >>>> That is, m3_build calls fold_build. >>>> >>>> - Jay >>>> >>>> >>>> >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 9 15:25:48 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 09 Mar 2010 15:25:48 +0100 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT Message-ID: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com> some windows problems again, probably only help needed... any takers? Olaf ----- Forwarded message from bugs at elego.de ----- Date: Tue, 09 Mar 2010 02:55:44 -0000 From: CM3 Reply-To: CM3 Subject: [CM3] #1082: Windows NT To: @MISSING_DOMAIN #1082: Windows NT -------------------------------------+-------------------------------------- Reporter: ilikesci@? | Owner: wagner Type: sw-test | Status: new Priority: medium | Milestone: Component: misc | Version: 5.8-RC3 Severity: non-critical | Keywords: Relnote: | Org: Estimatedhours: 0 | Hours: 0 Billable: 0 | Totalhours: 0 Internal: 0 | -------------------------------------+-------------------------------------- Htr: Remove msys out of your path. Fix: Env: Microsoft Windows Vista -------------------------------------+-------------------------------------- I am new to modula-3 but have tried a few times to get cm3 working on my MS-Windows machines over the years. This is my new attempt at that. I downloaded the NT386 files and have them unpacked. I have put cm3 in e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is that when I ran cminstall it asked for a tar.exe, gzip.exe, and msys-1.0.dll. I just happened to have msys installed on my computer and dropped them in the folder and cminstall seemed to work. I just wanted to let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and so I ran it per the suggestion on your website. It asked a few questions about what browser I wanted to use and such. The browser does pop up on localhost:3800 but times out. The console spits out: Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 calling start_browser(http://localhost:3800/) starting TCP service start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" http://localhost:3800/ CM3-IDE is shutting down because start_browser() returned TRUE. TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept TCPServer: aborting... I am thinking maybe something else might be running on the port and causing a problem loading page. I copied the startReactor.cmd to the bin directory and tried it and got the following: ------------------------------------------------------------------------------- startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by RCC ------------------------------------------------------------------------------- FATAL ERROR: Unable to find CM3 installation. CM3_ROOT expected in folder C:\cm3 CM3_BIN expected in folder C:\cm3\bin CM3.EXE expected in file C:\cm3\bin\cm3.exe Reactor.EXE expected in file C:\cm3\bin\reactor.exe cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD ------------------------------------------------------------------------------- Which it is installed on e: instead of c: but there is not a reactor.exe file in the bin directory either. From that information I am thinking I should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as e:\cm3. This is FYI and will let you know if I continue and any other experiences that might be of use. Thank You, Micah -- Ticket URL: CM3 Critical Mass Modula3 Compiler ----- End forwarded message ----- -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 9 16:29:58 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 15:29:58 +0000 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT In-Reply-To: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com> References: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com> Message-ID: Try the .msi instead. Try this maybe, I can't find the link on the web page (maybe lost in the crash?) http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/www/installation-windows.html?rev=1.5;content-type=text%2Fplain I should edit/rewrite those. - The bit about "upgrade is pessmistic" isn't true, as my Russian friend says, it is "realistic". There are several breaking changes that motivate doing it the way upgrade does it. - I should steer people to Python instead of cmd (really). - Mentions of 5.2.6 and 5.5.0 and maybe even building from source should probably go away. Leave this for "more beginners". - I need to update the Visual C++ redistributable link (9.0SP1 vs. 9.0). - The config file path is wrong. It is not config-no-install instead of config. - It alludes to too many options, using various versions, building from source or not, etc. But it seems not too bad. It should be retested. The IDE is not what you would normally call an IDE. It is custom web server. Seems like "apples and oranges", but it does make some sense. In some ways it was ahead of its time -- people talk about "application servers" these days, and run C# in the web server to produce html. Just not clear what sorts of apps are suited to this model. There's no editor -- not a bad idea, since everyone prefers what they already have. There's no debugger -- not a bad idea, not like we have the time to write another. You can use Visual Studio or windbg. There is a bit of a project system. However Modula-3 has among the best text file driven command line build systems around, we just need it to handle walking directories, and have it determine the order. It does provide hyperlinking built source, though personally I just use my editor's "find in files" all the time. Maybe someone else can write up something to replace. http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/www/installation-windows.html?rev=1.5;content-type=text%2Fplain#buildm3current Fixing the .cmd to find the .exes should be trivial. Or just have user run the .exes. Have the .exes set any environment variables they need. - Jay > Date: Tue, 9 Mar 2010 15:25:48 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: [M3devel] Fwd: [CM3] #1082: Windows NT > > some windows problems again, probably only help needed... > > any takers? > > Olaf > > ----- Forwarded message from bugs at elego.de ----- > Date: Tue, 09 Mar 2010 02:55:44 -0000 > From: CM3 > Reply-To: CM3 > Subject: [CM3] #1082: Windows NT > To: @MISSING_DOMAIN > > #1082: Windows NT > -------------------------------------+-------------------------------------- > Reporter: ilikesci@? | Owner: wagner > Type: sw-test | Status: new > Priority: medium | Milestone: > Component: misc | Version: 5.8-RC3 > Severity: non-critical | Keywords: > Relnote: | Org: > Estimatedhours: 0 | Hours: 0 > Billable: 0 | Totalhours: 0 > Internal: 0 | > -------------------------------------+-------------------------------------- > Htr: > Remove msys out of your path. > > > Fix: > > > > Env: > Microsoft Windows Vista > > -------------------------------------+-------------------------------------- > I am new to modula-3 but have tried a few times to get cm3 working on my > MS-Windows machines over the years. This is my new attempt at that. I > downloaded the NT386 files and have them unpacked. I have put cm3 in > e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is > that when I ran cminstall it asked for a tar.exe, gzip.exe, and > msys-1.0.dll. I just happened to have msys installed on my computer and > dropped them in the folder and cminstall seemed to work. I just wanted to > let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and > so I ran it per the suggestion on your website. It asked a few questions > about what browser I wanted to use and such. The browser does pop up on > localhost:3800 but times out. The console spits out: > > Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 > calling start_browser(http://localhost:3800/) > starting TCP service > start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" > http://localhost:3800/ > CM3-IDE is shutting down because start_browser() returned TRUE. > TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept > TCPServer: aborting... > > I am thinking maybe something else might be running on the port and > causing a problem loading page. I copied the startReactor.cmd to the bin > directory and tried it and got the following: > > > ------------------------------------------------------------------------------- > startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by > RCC > > ------------------------------------------------------------------------------- > FATAL ERROR: Unable to find CM3 installation. > CM3_ROOT expected in folder C:\cm3 > CM3_BIN expected in folder C:\cm3\bin > CM3.EXE expected in file C:\cm3\bin\cm3.exe > Reactor.EXE expected in file C:\cm3\bin\reactor.exe > cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD > > ------------------------------------------------------------------------------- > Which it is installed on e: instead of c: but there is not a reactor.exe > file in the bin directory either. From that information I am thinking I > should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as > e:\cm3. > > This is FYI and will let you know if I continue and any other experiences > that might be of use. > > Thank You, > Micah > > -- > Ticket URL: > CM3 > Critical Mass Modula3 Compiler > > > ----- End forwarded message ----- > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 9 16:36:18 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 15:36:18 +0000 Subject: [M3devel] 386?486?586?686?etc.? In-Reply-To: References: , , <20100208035847.GB13743@topoi.pooq.com>, Message-ID: > I'm still running an old 100 MHz Pentium and using it on a daily basis. There was a recent thread on the gcc lists about this. Fedora 11 requires Pentium ("586"). Fedora 12 requires Pentium Pro ("686"). Various distros also use xz/lzma instead of gzip. But OpenBSD sticks with gzip for perf on old machines. Our .deb files use lzma, if the machine building them has the tools. >> Wow. What for? And with Modula-3? What OS? - Jay From: jay.krell at cornell.edu To: hendrik at topoi.pooq.com; m3devel at elegosoft.com Date: Mon, 8 Feb 2010 15:22:21 +0000 Subject: Re: [M3devel] 386?486?586?686?etc.? Wow. What for? And with Modula-3? What OS? I think Pentium will be ok. I think ultimately, if people really need, we should have separate targets. As I've been saying, like: I386_FREEBSD_USERTHREADS, I586_FREEBSD, etc. Esp. to enable easier "release engineering", such as when we do more cross builds, adding new targets will be cheaper. But we'd want some sort of system where if nobody downloads and installs and minimally tests a release, it is in some low grade classification. Certain ones we'd test automatically, like whatever we have available in Hudson. -Jay > Date: Sun, 7 Feb 2010 22:58:48 -0500 > From: hendrik at topoi.pooq.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] 386?486?586?686?etc.? > > On Sun, Feb 07, 2010 at 11:59:11PM +0000, Jay K wrote: > > > > Any opinions/counter-opinions on which processors we should support? > > > > Presumably it doesn't vary per platform. > > > > Like, it wouldn't be Linux/586 and FreeBSD/486 or such. > > > > Unless maybe there is clear data about what the kernels support? > > > > The atomic stuff is pushing things to i586. > > I believe before 486 and possibly 386 worked. > > > > 686 is probably reasonable. > > > > I think it is Pentium II or Pentium Pro or newer, stuff like 15 years old already. > > I'm still running an old 100 MHz Pentium and using it on a daily basis. > > Debian has dropped support for the 386 with, as far as I know, no > complaints. > > -- hendrik. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 9 16:53:25 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 15:53:25 +0000 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT In-Reply-To: References: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com>, Message-ID: Sadly the .msis don't appear on the download page. Perhaps they are there but not indexed? Not that I can tell. There are some here: http://modula3.elegosoft.com/cm3/uploaded-archives/ I'll try them out and make some new ones, from the release branch. Last I tried, and probably currently, there is an unfortunate strict matching requirement between our build and what version of Visual C++ you use. Therefore it behooves us to e.g. provide Visual C++ 8.0 and 9.0 builds. I suggested we double up the Hudson NT386 tasks in that way. I can upload pairs (or more) of .msis, but we are sort of supposed to not take "random developer builds" but use "official package builds". I find the fact that we produce 10+ packages per platform..confusing, overwhelming. I wish there was just one or perhaps two, even though they'd be large. I understand people like to factor things into small pieces, but monolithism has its upsides. - Jay From: jay.krell at cornell.edu To: wagner at elegosoft.com; m3devel at elegosoft.com Date: Tue, 9 Mar 2010 15:29:58 +0000 Subject: Re: [M3devel] Fwd: [CM3] #1082: Windows NT Try the .msi instead. Try this maybe, I can't find the link on the web page (maybe lost in the crash?) http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/www/installation-windows.html?rev=1.5;content-type=text%2Fplain I should edit/rewrite those. - The bit about "upgrade is pessmistic" isn't true, as my Russian friend says, it is "realistic". There are several breaking changes that motivate doing it the way upgrade does it. - I should steer people to Python instead of cmd (really). - Mentions of 5.2.6 and 5.5.0 and maybe even building from source should probably go away. Leave this for "more beginners". - I need to update the Visual C++ redistributable link (9.0SP1 vs. 9.0). - The config file path is wrong. It is not config-no-install instead of config. - It alludes to too many options, using various versions, building from source or not, etc. But it seems not too bad. It should be retested. The IDE is not what you would normally call an IDE. It is custom web server. Seems like "apples and oranges", but it does make some sense. In some ways it was ahead of its time -- people talk about "application servers" these days, and run C# in the web server to produce html. Just not clear what sorts of apps are suited to this model. There's no editor -- not a bad idea, since everyone prefers what they already have. There's no debugger -- not a bad idea, not like we have the time to write another. You can use Visual Studio or windbg. There is a bit of a project system. However Modula-3 has among the best text file driven command line build systems around, we just need it to handle walking directories, and have it determine the order. It does provide hyperlinking built source, though personally I just use my editor's "find in files" all the time. Maybe someone else can write up something to replace. http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/www/installation-windows.html?rev=1.5;content-type=text%2Fplain#buildm3current Fixing the .cmd to find the .exes should be trivial. Or just have user run the .exes. Have the .exes set any environment variables they need. - Jay > Date: Tue, 9 Mar 2010 15:25:48 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: [M3devel] Fwd: [CM3] #1082: Windows NT > > some windows problems again, probably only help needed... > > any takers? > > Olaf > > ----- Forwarded message from bugs at elego.de ----- > Date: Tue, 09 Mar 2010 02:55:44 -0000 > From: CM3 > Reply-To: CM3 > Subject: [CM3] #1082: Windows NT > To: @MISSING_DOMAIN > > #1082: Windows NT > -------------------------------------+-------------------------------------- > Reporter: ilikesci@? | Owner: wagner > Type: sw-test | Status: new > Priority: medium | Milestone: > Component: misc | Version: 5.8-RC3 > Severity: non-critical | Keywords: > Relnote: | Org: > Estimatedhours: 0 | Hours: 0 > Billable: 0 | Totalhours: 0 > Internal: 0 | > -------------------------------------+-------------------------------------- > Htr: > Remove msys out of your path. > > > Fix: > > > > Env: > Microsoft Windows Vista > > -------------------------------------+-------------------------------------- > I am new to modula-3 but have tried a few times to get cm3 working on my > MS-Windows machines over the years. This is my new attempt at that. I > downloaded the NT386 files and have them unpacked. I have put cm3 in > e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is > that when I ran cminstall it asked for a tar.exe, gzip.exe, and > msys-1.0.dll. I just happened to have msys installed on my computer and > dropped them in the folder and cminstall seemed to work. I just wanted to > let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and > so I ran it per the suggestion on your website. It asked a few questions > about what browser I wanted to use and such. The browser does pop up on > localhost:3800 but times out. The console spits out: > > Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 > calling start_browser(http://localhost:3800/) > starting TCP service > start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" > http://localhost:3800/ > CM3-IDE is shutting down because start_browser() returned TRUE. > TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept > TCPServer: aborting... > > I am thinking maybe something else might be running on the port and > causing a problem loading page. I copied the startReactor.cmd to the bin > directory and tried it and got the following: > > > ------------------------------------------------------------------------------- > startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by > RCC > > ------------------------------------------------------------------------------- > FATAL ERROR: Unable to find CM3 installation. > CM3_ROOT expected in folder C:\cm3 > CM3_BIN expected in folder C:\cm3\bin > CM3.EXE expected in file C:\cm3\bin\cm3.exe > Reactor.EXE expected in file C:\cm3\bin\reactor.exe > cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD > > ------------------------------------------------------------------------------- > Which it is installed on e: instead of c: but there is not a reactor.exe > file in the bin directory either. From that information I am thinking I > should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as > e:\cm3. > > This is FYI and will let you know if I continue and any other experiences > that might be of use. > > Thank You, > Micah > > -- > Ticket URL: > CM3 > Critical Mass Modula3 Compiler > > > ----- End forwarded message ----- > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 9 17:46:52 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 16:46:52 +0000 Subject: [M3devel] PPC64_DARWIN Message-ID: Fyi, PPC64_DARWIN mostly works. "Ship" has a strong tendency to hang. Whenever I've looked, the stack has no symbols. I tried growing jmpbuf, using default stack, no luck yet. Will look more "later". Anyone running a Mac/G5? It's a big endian 64bit platform, if that is interesting. :) (SPARC64_SOLARIS will be too.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Tue Mar 9 15:43:44 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Tue, 9 Mar 2010 09:43:44 -0500 Subject: [M3devel] 386?486?586?686?etc.? In-Reply-To: References: Message-ID: <20100309144344.GC12734@topoi.pooq.com> On Tue, Mar 09, 2010 at 03:36:18PM +0000, Jay K wrote: > > > I'm still running an old 100 MHz Pentium and using it on a daily basis. > >> Wow. What for? And with Modula-3? What OS? It's my most reliable machine. The kind of reliable that's been running for something like two decades and shows signs of running for two more if I can continue to get secure OS support for it. My newer machines keep burning up graphics cards and the like. It is running Debian Lenny at the moment; a few months ago it was on Debian Etch. It's the boundary between my LAN and the internet. I last used Modula 3 on it a few years ago, throwing together an experimental type-checked language interpreter to test some ideas about language design. I'm not running Modula 3 on it right now, but since Modula 3 is still my systems language of choice, I'd hesitate to say I won't be doing it again. - hendrik > > > - Jay > > > > > > From: jay.krell at cornell.edu > To: hendrik at topoi.pooq.com; m3devel at elegosoft.com > Date: Mon, 8 Feb 2010 15:22:21 +0000 > Subject: Re: [M3devel] 386?486?586?686?etc.? > > > > Wow. What for? And with Modula-3? What OS? > I think Pentium will be ok. > I think ultimately, if people really need, we should have separate targets. > As I've been saying, like: I386_FREEBSD_USERTHREADS, I586_FREEBSD, etc. > Esp. to enable easier "release engineering", such as when we do more cross builds, > adding new targets will be cheaper. But we'd want some sort of system > where if nobody downloads and installs and minimally tests a release, it is > in some low grade classification. > Certain ones we'd test automatically, like whatever we have available in Hudson. > > -Jay > > > Date: Sun, 7 Feb 2010 22:58:48 -0500 > > From: hendrik at topoi.pooq.com > > To: m3devel at elegosoft.com > > Subject: Re: [M3devel] 386?486?586?686?etc.? > > > > On Sun, Feb 07, 2010 at 11:59:11PM +0000, Jay K wrote: > > > > > > Any opinions/counter-opinions on which processors we should support? > > > > > > Presumably it doesn't vary per platform. > > > > > > Like, it wouldn't be Linux/586 and FreeBSD/486 or such. > > > > > > Unless maybe there is clear data about what the kernels support? > > > > > > The atomic stuff is pushing things to i586. > > > I believe before 486 and possibly 386 worked. > > > > > > 686 is probably reasonable. > > > > > > I think it is Pentium II or Pentium Pro or newer, stuff like 15 years old already. > > > > I'm still running an old 100 MHz Pentium and using it on a daily basis. > > > > Debian has dropped support for the 386 with, as far as I know, no > > complaints. > > > > -- hendrik. > From rcolebur at SCIRES.COM Wed Mar 10 03:10:57 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Tue, 9 Mar 2010 21:10:57 -0500 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT In-Reply-To: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com> References: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com> Message-ID: Olaf: The "startReactor.CMD" file is obsolete and no longer in the repository. The new files are in "scripts\install\windows" and should be copied to the "cm3\bin" folder on a windows installation. "cm3StartIDE.CMD" replaces "startReactor.CMD". This file makes use of "cm3CommandShell.CMD". It appears that the problem reporter has his installation rooted at E:\cm3 rather than C:\cm3. So, he will need to set an environment variable "set CM3_ROOT=E:\cm3" to let these scripts know where to find the cm3 installation. Alternately, cm3CommandShell can be invoked with the arguments "Root E:\cm3" to tell it where to find the installation. Regards, Randy -----Original Message----- From: Olaf Wagner [mailto:wagner at elegosoft.com] Sent: Tuesday, March 09, 2010 9:26 AM To: m3devel Subject: [M3devel] Fwd: [CM3] #1082: Windows NT some windows problems again, probably only help needed... any takers? Olaf ----- Forwarded message from bugs at elego.de ----- Date: Tue, 09 Mar 2010 02:55:44 -0000 From: CM3 Reply-To: CM3 Subject: [CM3] #1082: Windows NT To: @MISSING_DOMAIN #1082: Windows NT -------------------------------------+-------------------------------------- Reporter: ilikesci@? | Owner: wagner Type: sw-test | Status: new Priority: medium | Milestone: Component: misc | Version: 5.8-RC3 Severity: non-critical | Keywords: Relnote: | Org: Estimatedhours: 0 | Hours: 0 Billable: 0 | Totalhours: 0 Internal: 0 | -------------------------------------+-------------------------------------- Htr: Remove msys out of your path. Fix: Env: Microsoft Windows Vista -------------------------------------+-------------------------------------- I am new to modula-3 but have tried a few times to get cm3 working on my MS-Windows machines over the years. This is my new attempt at that. I downloaded the NT386 files and have them unpacked. I have put cm3 in e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is that when I ran cminstall it asked for a tar.exe, gzip.exe, and msys-1.0.dll. I just happened to have msys installed on my computer and dropped them in the folder and cminstall seemed to work. I just wanted to let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and so I ran it per the suggestion on your website. It asked a few questions about what browser I wanted to use and such. The browser does pop up on localhost:3800 but times out. The console spits out: Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 calling start_browser(http://localhost:3800/) starting TCP service start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" http://localhost:3800/ CM3-IDE is shutting down because start_browser() returned TRUE. TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept TCPServer: aborting... I am thinking maybe something else might be running on the port and causing a problem loading page. I copied the startReactor.cmd to the bin directory and tried it and got the following: ------------------------------------------------------------------------------- startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by RCC ------------------------------------------------------------------------------- FATAL ERROR: Unable to find CM3 installation. CM3_ROOT expected in folder C:\cm3 CM3_BIN expected in folder C:\cm3\bin CM3.EXE expected in file C:\cm3\bin\cm3.exe Reactor.EXE expected in file C:\cm3\bin\reactor.exe cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD ------------------------------------------------------------------------------- Which it is installed on e: instead of c: but there is not a reactor.exe file in the bin directory either. From that information I am thinking I should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as e:\cm3. This is FYI and will let you know if I continue and any other experiences that might be of use. Thank You, Micah -- Ticket URL: CM3 Critical Mass Modula3 Compiler ----- End forwarded message ----- -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. From jay.krell at cornell.edu Wed Mar 10 07:52:42 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 10 Mar 2010 06:52:42 +0000 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT In-Reply-To: References: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com>, Message-ID: %~dp0 is a cmd file's directory. If cmd and exe are next to each other, they can find each other. As well, what does the cmd do that it can't be done in the exe? - Jay (phone) > From: rcolebur at SCIRES.COM > To: wagner at elegosoft.com; m3devel at elegosoft.com > Date: Tue, 9 Mar 2010 21:10:57 -0500 > Subject: Re: [M3devel] Fwd: [CM3] #1082: Windows NT > > Olaf: > > The "startReactor.CMD" file is obsolete and no longer in the repository. > > The new files are in "scripts\install\windows" and should be copied to the "cm3\bin" folder on a windows installation. > > "cm3StartIDE.CMD" replaces "startReactor.CMD". This file makes use of "cm3CommandShell.CMD". > > It appears that the problem reporter has his installation rooted at E:\cm3 rather than C:\cm3. So, he will need to set an environment variable "set CM3_ROOT=E:\cm3" to let these scripts know where to find the cm3 installation. Alternately, cm3CommandShell can be invoked with the arguments "Root E:\cm3" to tell it where to find the installation. > > Regards, > Randy > > -----Original Message----- > From: Olaf Wagner [mailto:wagner at elegosoft.com] > Sent: Tuesday, March 09, 2010 9:26 AM > To: m3devel > Subject: [M3devel] Fwd: [CM3] #1082: Windows NT > > some windows problems again, probably only help needed... > > any takers? > > Olaf > > ----- Forwarded message from bugs at elego.de ----- > Date: Tue, 09 Mar 2010 02:55:44 -0000 > From: CM3 > Reply-To: CM3 > Subject: [CM3] #1082: Windows NT > To: @MISSING_DOMAIN > > #1082: Windows NT > -------------------------------------+-------------------------------------- > Reporter: ilikesci@? | Owner: wagner > Type: sw-test | Status: new > Priority: medium | Milestone: > Component: misc | Version: 5.8-RC3 > Severity: non-critical | Keywords: > Relnote: | Org: > Estimatedhours: 0 | Hours: 0 > Billable: 0 | Totalhours: 0 > Internal: 0 | > -------------------------------------+-------------------------------------- > Htr: > Remove msys out of your path. > > > Fix: > > > > Env: > Microsoft Windows Vista > > -------------------------------------+-------------------------------------- > I am new to modula-3 but have tried a few times to get cm3 working on my > MS-Windows machines over the years. This is my new attempt at that. I > downloaded the NT386 files and have them unpacked. I have put cm3 in > e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is > that when I ran cminstall it asked for a tar.exe, gzip.exe, and > msys-1.0.dll. I just happened to have msys installed on my computer and > dropped them in the folder and cminstall seemed to work. I just wanted to > let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and > so I ran it per the suggestion on your website. It asked a few questions > about what browser I wanted to use and such. The browser does pop up on > localhost:3800 but times out. The console spits out: > > Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 > calling start_browser(http://localhost:3800/) > starting TCP service > start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" > http://localhost:3800/ > CM3-IDE is shutting down because start_browser() returned TRUE. > TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept > TCPServer: aborting... > > I am thinking maybe something else might be running on the port and > causing a problem loading page. I copied the startReactor.cmd to the bin > directory and tried it and got the following: > > > ------------------------------------------------------------------------------- > startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by > RCC > > ------------------------------------------------------------------------------- > FATAL ERROR: Unable to find CM3 installation. > CM3_ROOT expected in folder C:\cm3 > CM3_BIN expected in folder C:\cm3\bin > CM3.EXE expected in file C:\cm3\bin\cm3.exe > Reactor.EXE expected in file C:\cm3\bin\reactor.exe > cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD > > ------------------------------------------------------------------------------- > Which it is installed on e: instead of c: but there is not a reactor.exe > file in the bin directory either. From that information I am thinking I > should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as > e:\cm3. > > This is FYI and will let you know if I continue and any other experiences > that might be of use. > > Thank You, > Micah > > -- > Ticket URL: > CM3 > Critical Mass Modula3 Compiler > > > ----- End forwarded message ----- > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > > > CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. > > EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Wed Mar 10 14:10:41 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Wed, 10 Mar 2010 14:10:41 +0100 Subject: [M3devel] [CM3] #1082: Windows NT Message-ID: <20100310141041.t3hp8s034ss0cggc@mail.elegosoft.com> Have you read the answers on the m3devel list to which I forwarded your report? If not, you will find them in the archives at https://mail.elegosoft.com/pipermail/m3devel/2010-March/thread.html Hope that helps, @Jay/Randy, you should add your information to the trac ticket, too: https://projects.elego.de/cm3/ticket/1082 Sorry that I cannot do much myself currently, Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From rcolebur at SCIRES.COM Wed Mar 10 16:35:29 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Wed, 10 Mar 2010 10:35:29 -0500 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT In-Reply-To: References: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com>, Message-ID: Jay, the CMD files are conveniences I've found useful under Windows. I can supply some .REG files that will allow you to integrate these into the Explorer shell so that you can start a cm3 command shell in any folder and you can start CM3IDE. These CMD files ensure the environment is set up properly, including Visual C++. I understand the use of %~dp0 , but I already have an optional command line argument to deal with different locations for cm3 installation. Nevertheless, I can build in some more search capability in the cmd files if desired. Perhaps using %~dp0 would be a good tactic; I'll look into adding it. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, March 10, 2010 1:53 AM To: Coleburn, Randy; Olaf Wagner; m3devel Subject: RE: [M3devel] Fwd: [CM3] #1082: Windows NT %~dp0 is a cmd file's directory. If cmd and exe are next to each other, they can find each other. As well, what does the cmd do that it can't be done in the exe? - Jay (phone) > From: rcolebur at SCIRES.COM > To: wagner at elegosoft.com; m3devel at elegosoft.com > Date: Tue, 9 Mar 2010 21:10:57 -0500 > Subject: Re: [M3devel] Fwd: [CM3] #1082: Windows NT > > Olaf: > > The "startReactor.CMD" file is obsolete and no longer in the repository. > > The new files are in "scripts\install\windows" and should be copied to the "cm3\bin" folder on a windows installation. > > "cm3StartIDE.CMD" replaces "startReactor.CMD". This file makes use of "cm3CommandShell.CMD". > > It appears that the problem reporter has his installation rooted at E:\cm3 rather than C:\cm3. So, he will need to set an environment variable "set CM3_ROOT=E:\cm3" to let these scripts know where to find the cm3 installation. Alternately, cm3CommandShell can be invoked with the arguments "Root E:\cm3" to tell it where to find the installation. > > Regards, > Randy > > -----Original Message----- > From: Olaf Wagner [mailto:wagner at elegosoft.com] > Sent: Tuesday, March 09, 2010 9:26 AM > To: m3devel > Subject: [M3devel] Fwd: [CM3] #1082: Windows NT > > some windows problems again, probably only help needed... > > any takers? > > Olaf > > ----- Forwarded message from bugs at elego.de ----- > Date: Tue, 09 Mar 2010 02:55:44 -0000 > From: CM3 > Reply-To: CM3 > Subject: [CM3] #1082: Windows NT > To: @MISSING_DOMAIN > > #1082: Windows NT > -------------------------------------+-------------------------------------- > Reporter: ilikesci at ... | Owner: wagner > Type: sw-test | Status: new > Priority: medium | Milestone: > Component: misc | Version: 5.8-RC3 > Severity: non-critical | Keywords: > Relnote: | Org: > Estimatedhours: 0 | Hours: 0 > Billable: 0 | Totalhours: 0 > Internal: 0 | > -------------------------------------+-------------------------------------- > Htr: > Remove msys out of your path. > > > Fix: > > > > Env: > Microsoft Windows Vista > > -------------------------------------+-------------------------------------- > I am new to modula-3 but have tried a few times to get cm3 working on my > MS-Windows machines over the years. This is my new attempt at that. I > downloaded the NT386 files and have them unpacked. I have put cm3 in > e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is > that when I ran cminstall it asked for a tar.exe, gzip.exe, and > msys-1.0.dll. I just happened to have msys installed on my computer and > dropped them in the folder and cminstall seemed to work. I just wanted to > let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and > so I ran it per the suggestion on your website. It asked a few questions > about what browser I wanted to use and such. The browser does pop up on > localhost:3800 but times out. The console spits out: > > Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 > calling start_browser(http://localhost:3800/) > starting TCP service > start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" > http://localhost:3800/ > CM3-IDE is shutting down because start_browser() returned TRUE. > TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept > TCPServer: aborting... > > I am thinking maybe something else might be running on the port and > causing a problem loading page. I copied the startReactor.cmd to the bin > directory and tried it and got the following: > > > ------------------------------------------------------------------------------- > startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by > RCC > > ------------------------------------------------------------------------------- > FATAL ERROR: Unable to find CM3 installation. > CM3_ROOT expected in folder C:\cm3 > CM3_BIN expected in folder C:\cm3\bin > CM3.EXE expected in file C:\cm3\bin\cm3.exe > Reactor.EXE expected in file C:\cm3\bin\reactor.exe > cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD > > ------------------------------------------------------------------------------- > Which it is installed on e: instead of c: but there is not a reactor.exe > file in the bin directory either. From that information I am thinking I > should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as > e:\cm3. > > This is FYI and will let you know if I continue and any other experiences > that might be of use. > > Thank You, > Micah > > -- > Ticket URL: > CM3 > Critical Mass Modula3 Compiler > > > ----- End forwarded message ----- > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > > > CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. > > EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 10 16:56:32 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 10 Mar 2010 15:56:32 +0000 Subject: [M3devel] NT386 64bit LONGINT sub range checks oops Message-ID: Just a note that subrange checking of 64bit LONGINT is broken on NT386. I just noticed. 64bit LONGINT should work to a very large extent though. This is the only problem I know of. I'll look into it soon. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 10 17:09:46 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 10 Mar 2010 16:09:46 +0000 Subject: [M3devel] NT386 64bit LONGINT sub range checks oops In-Reply-To: References: Message-ID: Hm. I'm just not sure actually. I'll have to test it. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Wed, 10 Mar 2010 15:56:32 +0000 Subject: [M3devel] NT386 64bit LONGINT sub range checks oops Just a note that subrange checking of 64bit LONGINT is broken on NT386. I just noticed. 64bit LONGINT should work to a very large extent though. This is the only problem I know of. I'll look into it soon. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 11 15:45:21 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 11 Mar 2010 14:45:21 +0000 Subject: [M3devel] interface long, inlining large integer code, etc. In-Reply-To: <60FB82E7-B3B3-45ED-973B-191F7992A4E2@cs.purdue.edu> References: <20100308122636.83CA12474001@birch.elegosoft.com>, , , , , , , , , , , , , <5ECDB225-60F1-4094-96EA-B7BB0EBE340A@cs.purdue.edu>, , <60FB82E7-B3B3-45ED-973B-191F7992A4E2@cs.purdue.edu> Message-ID: I was thinking a bit more about this. Basically it can already be done -- we know the name of the function are compiling. If the function is Long__Extract, Long__Times, etc., we can inline. If it isn't, we can assume the existance and interface of such functions, and call them. Instead of calling anything in hand.c. Maybe even give them custom calling conventions. The front end knows stuff about interface Word/Long. It seems reasonable for m3back to also. The open question however is that there is no place to "hang" inlined versions of signed operations, what you might call Integer__Extract, Integer__Divide, LongInt__Div, LongInt__Times, LongInt__Mod, etc. Maybe they should be in C:\dev2\cm3.2\m3-libs\m3core\src\types? Similarly open question would be the set functions. RTHooks.SetLT? RTSet.LT? C:\dev2\cm3.2\m3-libs\m3core\src\types\set? C:\dev2\cm3.2\m3-libs\m3core\src\types\bigset? (as opposed to word-sized set, besides that eq/ne are inlined by the frontend for "medium" sizes) Is it clear what my questions are? Let me try again. Consider Long.Rotate. Assume it takes a bunch of code. m3back could notice it is compiling Long__Rotate, and generate the code inline. Otherwise it can "know" such a function exists and generate a call to it. This is a reasonable seeming model that I just came up with. That does double the paths in m3back, granted. For several such m3cg operations, there is already a natural function to treat this way -- stuff in interface Long. However, for a few operations, there is not already a place -- signed operations and operations on large sets. As well, there is a presumed question/answer/non-answer: everything works today. But should it be changed? Is it better to reduce/eliminate hand.c and instead teach the Modula-3 code in m3back how to generate assembly? "Better" how? More efficient? Maybe. The factor here is that hand.c may or may not be optimized, but m3back operates in a mode where it always optimizes to some extent. It generally generates code that is significantly better than unoptimized C, and significantly worse than optimized C, depending -- it does pretty well, but it never inlines, but it only does a certain small class of smart things. It keeps values in registers somewhat, it doesn't eliminate dead stores, doesn't unroll loops. Another factor is the principle of having C code or not. We vehemently agree that C code is preferred in m3core/unix. But otherwise many people here kind of dislike or discourage C on principle/philosophical grounds. I'm not sure I agree, however by agreeing, I introduce an interesting technical challenge. :) Again, must we really call it interface long? Does that really "sound like": INTEGER is to Word as LONGINT is to x? LongWord maybe? - Jay > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > CC: m3commit at elegosoft.com > Subject: RE: [M3commit] CVS Update: cm3 > Date: Thu, 11 Mar 2010 04:10:40 +0000 > > > Long.LeftShift is not a great example. Sorry. > Long.Shift, Long.Rotate, Long.Times, LongDivide. > Those are or "could be" large. Currently the later > three all call functions and are therefore short. > > > Integer.Multiple, Divide, Remainder, similarly uncertain. > They call functions, the functions aren't short. > > > I was also thinking, some of these functions, esp. > 64bit shift/rotate could be shorter as loops. > But I think not worthwhile. > > > To be clear, even though "parse.c" "never" generates > function calls (except for set operations), the gcc > backend itself often does. The functions are in libgcc. > > > You know..I find it hard to decide about compiler generated > function calls. Definitely for size optimization they > can be a win. But I just somehow vaguely don't like them. > In many cases, the decision is not difficult. set_singleton > and set_member are good examples. They could be inlined > and be very small. But Shift, Rotate, Multiply, Divide, > it gets less clear. > > > But no matter how large an inline form, I think > when interface word/long maps directly to these operations, > they should be inlined. > > > And then if there are places that don't inline, they should > call the "instantiations" in interface word/long. > > > And then, furthermore, we should consider an interface > for "Integer.Divide/Multiple/Remainder"? > A place to park the possibly large code? > > > - Jay > > > ---------------------------------------- > > From: hosking at cs.purdue.edu > > Date: Wed, 10 Mar 2010 21:01:54 -0500 > > To: jay.krell at cornell.edu > > CC: m3commit at elegosoft.com > > Subject: Re: [M3commit] CVS Update: cm3 > > > > I am confused. The intention is that Long.LeftShift is always inlined wherever it is called. Is the code for it really that hairy? > > > > On 10 Mar 2010, at 20:30, Jay K wrote: > > > >> > >> If there exists an inline expansion, and it is large, but there is a function whose job it is to hold the expansion, whether it is inline or not, and the backend was informed of this function's name, and whether or not it was currently producing it, the backend could generate the inline expansion for Long__LeftShift, but otherwise generate calls to it. > >> > >> > >> That is, e.g. m3cg.left_shift(type := int64|word64) could chose to either call Long__LeftShift or generate the body of Long__LeftShift. Not based on some "is inlining profitable heuristic", but specifically if told it is generating the function Long__LeftShift or not. > >> > >> > >> That is, there's no point in having a C function m3_left_shift64 or somesuch, and having Long__LeftShift call it. Instead a backend should generate the body of Long__LeftShift inline when told it is generating that function, vs. generating a call to that function when it is otherwise asked to do a 64bit leftshift if it decides that inlining it every time is too large. > >> > >> Granted at least two things: > >> Given 64bit target, Long__LeftShift vs. Word__LeftShift is ambiguous. > >> And I'm inlined to just always inline anyway. > >> > >> > >> (I still don't like the term "Long" here. It doesn't convey unsigned.) > >> > >> > >> - Jay > >> > >> > >> > >> ________________________________ > >>> Subject: Re: [M3commit] CVS Update: cm3 > >>> From: hosking at cs.purdue.edu > >>> Date: Wed, 10 Mar 2010 19:57:17 -0500 > >>> CC: m3commit at elegosoft.com > >>> To: jay.krell at cornell.edu > >>> > >>> > >>> > >>> Yes, someone can pass the function as a parameter. > >>> > >>> I don't understand the rest of what you are saying. > >>> > >>> > >>> Antony Hosking | Associate Professor | Computer Science | Purdue University > >>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>> > >>> > >>> > >>> > >>> > >>> > >>> On 10 Mar 2010, at 17:27, Jay K wrote: > >>> > >>> > >>> Hey, should we maybe have a flag that indicates we are producing a function that is meant to provide and only provide exactly the functionality of the "one" m3cg call? > >>> You know, that'd be a great hint to: > >>> > >>> - definitely definitely definitely definitely inline > >>> - if we know this thing existed, we could use it instead of a .c file. > >>> - along with the function's name for the invocations that aren't the function? > >>> > >>> > >>> Why provide the function anyway? In case someone takes the address? > >>> > >>> > >>> > >>> - Jay > >>> > >>> > >>> ________________________________ > >>> From: jay.krell at cornell.edu > >>> To: hosking at cs.purdue.edu > >>> Date: Mon, 8 Mar 2010 16:04:45 +0000 > >>> CC: m3commit at elegosoft.com > >>> Subject: Re: [M3commit] CVS Update: cm3 > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> Fair enough. > >>> > >>> > >>> > >>> - Jay > >>> > >>> > >>> > >>> ________________________________ > >>> > >>> From: hosking at cs.purdue.edu > >>> Date: Mon, 8 Mar 2010 10:29:00 -0500 > >>> To: jay.krell at cornell.edu > >>> CC: m3commit at elegosoft.com > >>> Subject: Re: [M3commit] CVS Update: cm3 > >>> > >>> > >>> > >>> > >>> You can't call the support function Long__Extract because we already have a Long__Extract in m3core/src/word. It is the library representative of the front-end compiler-generated (inlined) Long.Extract. > >>> > >>> > >>> > >>> On 8 Mar 2010, at 07:30, Jay K wrote: > >>> > >>> > >>> > >>> diff attached > >>> > >>> > >>> If folks really want to use tables or function calls here, let me know. > >>> > >>> > >>> The data was historically problematic, though we worked out the problems. > >>> - It was initialized at runtime which has a race condition, fixed years ago. > >>> - It can't be "exported" on NT386 (which is the only platform that uses it) and must be statically linked. > >>> > >>> The data is still used by Word.Insert and Long.Insert is still a function, but those > >>> are my next targets. > >>> > >>> > >>> This is a case where the user does write a function call. Word.Extract or Long.Extract. > >>> (So the function should have been called Long__Extract.) > >>> > >>> > >>> - Jay > >>> > >>> Date: Mon, 8 Mar 2010 13:26:36 +0000 > >>> To: m3commit at elegosoft.com > >>> From: jkrell at elego.de > >>> Subject: [M3commit] CVS Update: cm3 > >>> > >>> CVSROOT: /usr/cvs > >>> Changes by: jkrell at birch. 10/03/08 13:26:36 > >>> > >>> Modified files: > >>> cm3/m3-sys/m3back/src/: M3x86.m3 Stackx86.i3 Stackx86.m3 > >>> > >>> Log message: > >>> "rewrite" extract to not use tables and always be inlined for 64bit > >>> > >>> equivalent C code: > >>> > >>> UT __stdcall extract(UT x, uint32 offset, uint32 count) > >>> { > >>> x>>= offset; > >>> x &= ~((~(UT)0) << count); > >>> return x; > >>> } > >>> > >>> extract32 > >>> 00000729: 8B4DEC MOV ECX tv.126[_m] > >>> 0000072C: 8B5DF4 MOV EBX tv.124[_a32] > >>> 0000072F: D3EB SHR EBX > >>> 00000731: 8BD1 MOV EDX ECX This is pointless and I'll try to remove. > >>> 00000733: 8B4DE4 MOV ECX tv.128[_n] > >>> 00000736: BEFFFFFFFF MOV ESI $4294967295 I'll look for smaller form of this. or esi -1? > >>> 0000073B: D3E6 SHL ESI > >>> 0000073D: F7D6 NOT ESI > >>> 0000073F: 23DE AND EBX ESI > >>> > >>> extract64 > >>> 000008E4: 8B4DD8 MOV ECX tv.134[_m] > >>> 000008E7: 8B5DE8 MOV EBX tv.131[_a64] > >>> 000008EA: 8B55EC MOV EDX tv.131[_a64]+4 > >>> 000008ED: 0FADD3 SHRD EBX EDX ECX > >>> 000008F0: D3EA SHR EDX > >>> 000008F2: F6C120 TEST ECX $32 > >>> 000008F5: 7400 JE rel8 L.107 > >>> 000008F7: 8BDA MOV EBX EDX > >>> 000008F9: 33D2 XOR EDX EDX > >>> set_label L.107 > >>> 000008FB: 8BF1 MOV ESI ECX This is pointless and I'll try to remove. > >>> 000008FD: 8B4DD0 MOV ECX tv.136[_n] > >>> 00000900: BFFFFFFFFF MOV EDI $4294967295 I'll look for smaller form of this. or edi -1? > >>> 00000905: B8FFFFFFFF MOV EAX $4294967295 I'll look for smaller form of this. (heck, at least mov edi, eax) > >>> 0000090A: 0FA5F8 SHLD EAX EDI ECX > >>> 0000090D: D3E7 SHL EDI > >>> 0000090F: F6C120 TEST ECX $32 > >>> 00000912: 7400 JE rel8 L.108 > >>> 00000914: 8BC7 MOV EAX EDI > >>> 00000916: 33FF XOR EDI EDI > >>> set_label L.108 > >>> 00000918: F7D7 NOT EDI > >>> 0000091A: F7D0 NOT EAX > >>> 0000091C: 23DF AND EBX EDI > >>> 0000091E: 23D0 AND EDX EAX > >>> > >>> having n or m and n (or just m? I think so.) be constant leads to better code > >>> > >>> some other small cleanup, like avoiding calling find twice, > >>> I don't see why it was that way > >>> > >>> <1.txt> > >>> > >>> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 11 16:40:34 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 11 Mar 2010 15:40:34 +0000 Subject: [M3devel] Target.EOL Message-ID: I strongly suspect that Target.EOL can go away and just use Wr.EOL instead. Or even just \n. Cross builds are fairly rare, esp. cross between NT and Posix, and very many tools treat \n, \r\n, and perhaps \r the same, so even crossing NT <=> Posix doesn't likely matter. ie. it works, assuming you have a cross m3cg (ie: mingwin/cygwin hosted for NT hosts) gcc, Visual C++ compiler, m3front, all treat \r and \r\n the same. (witness all our *.h, *.c, *.i3, *.m3 files use \r\n) I think we have some tools that don't understand \r\n, but in my opinion that is a bug. All three formats should be treated the same. I know notepad doesn't display \n well, and Sun cc might not like \r\n. I know some compiler I used recently was picky, but "many" compilers are not. cmd might be ok with \n. Python calls it "universal newlines", treating all three formats the same. C:\dev2\cm3.2\m3-sys\cm3\src\Utils.m3(190):PROCEDURE CopyText (old, new: TEXT) = C:\dev2\cm3.2\m3-sys\m3middle\src\M3File.m3(61):PROCEDURE CopyText (src, dest: TEXT; eol: TEXT) RAISES {OSError.E} = probably leave alone, except that they don't treat \r correctly if you consider the historical Apple behavior. There are three formats, not two. Anyway, it's not a big deal. I think my problem here is solved by the change to m3x86.m3 I already made. I'll put Target.EOL back on my machine. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 11 18:31:42 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 11 Mar 2010 12:31:42 -0500 Subject: [M3devel] Target.EOL In-Reply-To: References: Message-ID: <6A2F1518-1C9C-4666-BC3E-B6B8D397A9EA@cs.purdue.edu> Sounds reasonable. On 11 Mar 2010, at 10:40, Jay K wrote: > I strongly suspect that Target.EOL can go away and just use Wr.EOL instead. > Or even just \n. > > > Cross builds are fairly rare, esp. cross between NT and Posix, and very > many tools treat \n, \r\n, and perhaps \r the same, so even > crossing NT <=> Posix doesn't likely matter. > ie. it works, assuming you have a cross m3cg (ie: mingwin/cygwin hosted for NT hosts) > > > gcc, Visual C++ compiler, m3front, all treat \r and \r\n the same. > (witness all our *.h, *.c, *.i3, *.m3 files use \r\n) > I think we have some tools that don't understand \r\n, but in my opinion that is a bug. > All three formats should be treated the same. > > > I know notepad doesn't display \n well, and Sun cc might not like \r\n. > I know some compiler I used recently was picky, but "many" compilers are not. > cmd might be ok with \n. > Python calls it "universal newlines", treating all three formats the same. > > > C:\dev2\cm3.2\m3-sys\cm3\src\Utils.m3(190):PROCEDURE CopyText (old, new: TEXT) = > C:\dev2\cm3.2\m3-sys\m3middle\src\M3File.m3(61):PROCEDURE CopyText (src, dest: TEXT; eol: TEXT) RAISES {OSError.E} = > > > probably leave alone, except that they don't treat \r correctly if you consider the historical Apple behavior. > There are three formats, not two. > > > Anyway, it's not a big deal. I think my problem here is solved by the change to m3x86.m3 I already made. > I'll put Target.EOL back on my machine. > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 11 19:03:43 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 11 Mar 2010 13:03:43 -0500 Subject: [M3devel] interface long, inlining large integer code, etc. In-Reply-To: References: <20100308122636.83CA12474001@birch.elegosoft.com>, , , , , , , , , , , , , <5ECDB225-60F1-4094-96EA-B7BB0EBE340A@cs.purdue.edu>, , <60FB82E7-B3B3-45ED-973B-191F7992A4E2@cs.purdue.edu> Message-ID: <8536088B-FFE4-43CA-AC63-442D66AE72A0@cs.purdue.edu> I'm going to resist this one. It's not clear that it matters terribly much whether these utility routines are inlined or not. If it is shown to be a performance bottleneck then perhaps. But I suspect that there are very few programs that pass the procedures of the Word/Long interface around by value. On 11 Mar 2010, at 09:45, Jay K wrote: > I was thinking a bit more about this. > Basically it can already be done -- we know the name of the function are compiling. > If the function is Long__Extract, Long__Times, etc., we can inline. > If it isn't, we can assume the existance and interface of such functions, and call them. > Instead of calling anything in hand.c. > Maybe even give them custom calling conventions. > The front end knows stuff about interface Word/Long. It seems reasonable for m3back to also. > > > The open question however is that there is no place to "hang" inlined versions of signed operations, what you might call Integer__Extract, Integer__Divide, LongInt__Div, LongInt__Times, LongInt__Mod, etc. > > > Maybe they should be in C:\dev2\cm3.2\m3-libs\m3core\src\types? > > > Similarly open question would be the set functions. > RTHooks.SetLT? > RTSet.LT? > C:\dev2\cm3.2\m3-libs\m3core\src\types\set? > C:\dev2\cm3.2\m3-libs\m3core\src\types\bigset? > (as opposed to word-sized set, besides that eq/ne are inlined by the frontend for "medium" sizes) > > > Is it clear what my questions are? > > > Let me try again. > Consider Long.Rotate. Assume it takes a bunch of code. > m3back could notice it is compiling Long__Rotate, and generate the code inline. > Otherwise it can "know" such a function exists and generate a call to it. > This is a reasonable seeming model that I just came up with. > > > That does double the paths in m3back, granted. > > For several such m3cg operations, there is already a natural function to treat this way -- stuff in interface Long. > > > However, for a few operations, there is not already a place -- signed operations and operations on large sets. > > > As well, there is a presumed question/answer/non-answer: everything works today. > But should it be changed? > Is it better to reduce/eliminate hand.c and instead teach the Modula-3 code in m3back how to generate assembly? > "Better" how? > More efficient? Maybe. The factor here is that hand.c may or may not be optimized, but m3back operates in a mode where it always optimizes to some extent. It generally generates code that is significantly better than unoptimized C, and significantly worse than optimized C, depending -- it does pretty well, but it never inlines, but it only does a certain small class of smart things. It keeps values in registers somewhat, it doesn't eliminate dead stores, doesn't unroll loops. > > > Another factor is the principle of having C code or not. > We vehemently agree that C code is preferred in m3core/unix. > But otherwise many people here kind of dislike or discourage C on principle/philosophical grounds. > I'm not sure I agree, however by agreeing, I introduce an interesting technical challenge. :) > > > Again, must we really call it interface long? > Does that really "sound like": > INTEGER is to Word as LONGINT is to x? > > LongWord maybe? > > > - Jay > > > > From: jay.krell at cornell.edu > > To: hosking at cs.purdue.edu > > CC: m3commit at elegosoft.com > > Subject: RE: [M3commit] CVS Update: cm3 > > Date: Thu, 11 Mar 2010 04:10:40 +0000 > > > > > > Long.LeftShift is not a great example. Sorry. > > Long.Shift, Long.Rotate, Long.Times, LongDivide. > > Those are or "could be" large. Currently the later > > three all call functions and are therefore short. > > > > > > Integer.Multiple, Divide, Remainder, similarly uncertain. > > They call functions, the functions aren't short. > > > > > > I was also thinking, some of these functions, esp. > > 64bit shift/rotate could be shorter as loops. > > But I think not worthwhile. > > > > > > To be clear, even though "parse.c" "never" generates > > function calls (except for set operations), the gcc > > backend itself often does. The functions are in libgcc. > > > > > > You know..I find it hard to decide about compiler generated > > function calls. Definitely for size optimization they > > can be a win. But I just somehow vaguely don't like them. > > In many cases, the decision is not difficult. set_singleton > > and set_member are good examples. They could be inlined > > and be very small. But Shift, Rotate, Multiply, Divide, > > it gets less clear. > > > > > > But no matter how large an inline form, I think > > when interface word/long maps directly to these operations, > > they should be inlined. > > > > > > And then if there are places that don't inline, they should > > call the "instantiations" in interface word/long. > > > > > > And then, furthermore, we should consider an interface > > for "Integer.Divide/Multiple/Remainder"? > > A place to park the possibly large code? > > > > > > - Jay > > > > > > ---------------------------------------- > > > From: hosking at cs.purdue.edu > > > Date: Wed, 10 Mar 2010 21:01:54 -0500 > > > To: jay.krell at cornell.edu > > > CC: m3commit at elegosoft.com > > > Subject: Re: [M3commit] CVS Update: cm3 > > > > > > I am confused. The intention is that Long.LeftShift is always inlined wherever it is called. Is the code for it really that hairy? > > > > > > On 10 Mar 2010, at 20:30, Jay K wrote: > > > > > >> > > >> If there exists an inline expansion, and it is large, but there is a function whose job it is to hold the expansion, whether it is inline or not, and the backend was informed of this function's name, and whether or not it was currently producing it, the backend could generate the inline expansion for Long__LeftShift, but otherwise generate calls to it. > > >> > > >> > > >> That is, e.g. m3cg.left_shift(type := int64|word64) could chose to either call Long__LeftShift or generate the body of Long__LeftShift. Not based on some "is inlining profitable heuristic", but specifically if told it is generating the function Long__LeftShift or not. > > >> > > >> > > >> That is, there's no point in having a C function m3_left_shift64 or somesuch, and having Long__LeftShift call it. Instead a backend should generate the body of Long__LeftShift inline when told it is generating that function, vs. generating a call to that function when it is otherwise asked to do a 64bit leftshift if it decides that inlining it every time is too large. > > >> > > >> Granted at least two things: > > >> Given 64bit target, Long__LeftShift vs. Word__LeftShift is ambiguous. > > >> And I'm inlined to just always inline anyway. > > >> > > >> > > >> (I still don't like the term "Long" here. It doesn't convey unsigned.) > > >> > > >> > > >> - Jay > > >> > > >> > > >> > > >> ________________________________ > > >>> Subject: Re: [M3commit] CVS Update: cm3 > > >>> From: hosking at cs.purdue.edu > > >>> Date: Wed, 10 Mar 2010 19:57:17 -0500 > > >>> CC: m3commit at elegosoft.com > > >>> To: jay.krell at cornell.edu > > >>> > > >>> > > >>> > > >>> Yes, someone can pass the function as a parameter. > > >>> > > >>> I don't understand the rest of what you are saying. > > >>> > > >>> > > >>> Antony Hosking | Associate Professor | Computer Science | Purdue University > > >>> 305 N. University Street | West Lafayette | IN 47907 | USA > > >>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> On 10 Mar 2010, at 17:27, Jay K wrote: > > >>> > > >>> > > >>> Hey, should we maybe have a flag that indicates we are producing a function that is meant to provide and only provide exactly the functionality of the "one" m3cg call? > > >>> You know, that'd be a great hint to: > > >>> > > >>> - definitely definitely definitely definitely inline > > >>> - if we know this thing existed, we could use it instead of a .c file. > > >>> - along with the function's name for the invocations that aren't the function? > > >>> > > >>> > > >>> Why provide the function anyway? In case someone takes the address? > > >>> > > >>> > > >>> > > >>> - Jay > > >>> > > >>> > > >>> ________________________________ > > >>> From: jay.krell at cornell.edu > > >>> To: hosking at cs.purdue.edu > > >>> Date: Mon, 8 Mar 2010 16:04:45 +0000 > > >>> CC: m3commit at elegosoft.com > > >>> Subject: Re: [M3commit] CVS Update: cm3 > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> Fair enough. > > >>> > > >>> > > >>> > > >>> - Jay > > >>> > > >>> > > >>> > > >>> ________________________________ > > >>> > > >>> From: hosking at cs.purdue.edu > > >>> Date: Mon, 8 Mar 2010 10:29:00 -0500 > > >>> To: jay.krell at cornell.edu > > >>> CC: m3commit at elegosoft.com > > >>> Subject: Re: [M3commit] CVS Update: cm3 > > >>> > > >>> > > >>> > > >>> > > >>> You can't call the support function Long__Extract because we already have a Long__Extract in m3core/src/word. It is the library representative of the front-end compiler-generated (inlined) Long.Extract. > > >>> > > >>> > > >>> > > >>> On 8 Mar 2010, at 07:30, Jay K wrote: > > >>> > > >>> > > >>> > > >>> diff attached > > >>> > > >>> > > >>> If folks really want to use tables or function calls here, let me know. > > >>> > > >>> > > >>> The data was historically problematic, though we worked out the problems. > > >>> - It was initialized at runtime which has a race condition, fixed years ago. > > >>> - It can't be "exported" on NT386 (which is the only platform that uses it) and must be statically linked. > > >>> > > >>> The data is still used by Word.Insert and Long.Insert is still a function, but those > > >>> are my next targets. > > >>> > > >>> > > >>> This is a case where the user does write a function call. Word.Extract or Long.Extract. > > >>> (So the function should have been called Long__Extract.) > > >>> > > >>> > > >>> - Jay > > >>> > > >>> Date: Mon, 8 Mar 2010 13:26:36 +0000 > > >>> To: m3commit at elegosoft.com > > >>> From: jkrell at elego.de > > >>> Subject: [M3commit] CVS Update: cm3 > > >>> > > >>> CVSROOT: /usr/cvs > > >>> Changes by: jkrell at birch. 10/03/08 13:26:36 > > >>> > > >>> Modified files: > > >>> cm3/m3-sys/m3back/src/: M3x86.m3 Stackx86.i3 Stackx86.m3 > > >>> > > >>> Log message: > > >>> "rewrite" extract to not use tables and always be inlined for 64bit > > >>> > > >>> equivalent C code: > > >>> > > >>> UT __stdcall extract(UT x, uint32 offset, uint32 count) > > >>> { > > >>> x>>= offset; > > >>> x &= ~((~(UT)0) << count); > > >>> return x; > > >>> } > > >>> > > >>> extract32 > > >>> 00000729: 8B4DEC MOV ECX tv.126[_m] > > >>> 0000072C: 8B5DF4 MOV EBX tv.124[_a32] > > >>> 0000072F: D3EB SHR EBX > > >>> 00000731: 8BD1 MOV EDX ECX This is pointless and I'll try to remove. > > >>> 00000733: 8B4DE4 MOV ECX tv.128[_n] > > >>> 00000736: BEFFFFFFFF MOV ESI $4294967295 I'll look for smaller form of this. or esi -1? > > >>> 0000073B: D3E6 SHL ESI > > >>> 0000073D: F7D6 NOT ESI > > >>> 0000073F: 23DE AND EBX ESI > > >>> > > >>> extract64 > > >>> 000008E4: 8B4DD8 MOV ECX tv.134[_m] > > >>> 000008E7: 8B5DE8 MOV EBX tv.131[_a64] > > >>> 000008EA: 8B55EC MOV EDX tv.131[_a64]+4 > > >>> 000008ED: 0FADD3 SHRD EBX EDX ECX > > >>> 000008F0: D3EA SHR EDX > > >>> 000008F2: F6C120 TEST ECX $32 > > >>> 000008F5: 7400 JE rel8 L.107 > > >>> 000008F7: 8BDA MOV EBX EDX > > >>> 000008F9: 33D2 XOR EDX EDX > > >>> set_label L.107 > > >>> 000008FB: 8BF1 MOV ESI ECX This is pointless and I'll try to remove. > > >>> 000008FD: 8B4DD0 MOV ECX tv.136[_n] > > >>> 00000900: BFFFFFFFFF MOV EDI $4294967295 I'll look for smaller form of this. or edi -1? > > >>> 00000905: B8FFFFFFFF MOV EAX $4294967295 I'll look for smaller form of this. (heck, at least mov edi, eax) > > >>> 0000090A: 0FA5F8 SHLD EAX EDI ECX > > >>> 0000090D: D3E7 SHL EDI > > >>> 0000090F: F6C120 TEST ECX $32 > > >>> 00000912: 7400 JE rel8 L.108 > > >>> 00000914: 8BC7 MOV EAX EDI > > >>> 00000916: 33FF XOR EDI EDI > > >>> set_label L.108 > > >>> 00000918: F7D7 NOT EDI > > >>> 0000091A: F7D0 NOT EAX > > >>> 0000091C: 23DF AND EBX EDI > > >>> 0000091E: 23D0 AND EDX EAX > > >>> > > >>> having n or m and n (or just m? I think so.) be constant leads to better code > > >>> > > >>> some other small cleanup, like avoiding calling find twice, > > >>> I don't see why it was that way > > >>> > > >>> <1.txt> > > >>> > > >>> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Thu Mar 11 23:03:30 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 17:03:30 -0500 Subject: [M3devel] Target.EOL In-Reply-To: References: Message-ID: I don't think I've used Target.EOL, but I've definitely used some EOL definition somewhere, maybe it was Wr.EOL, I'd have to check to be sure. I'm not sure the implications of removing Target.EOL, but whatever is done, we need to maintain a way to distinguish at runtime the proper line ending format for the host environment. I've written a lot of code in the past where this is important. Some of my code interfaces with other systems whose line ending formats have to be compared and/or reconciled with the host when doing serial data transfer (yes there are still systems that rely on good 'ol serial I/O, particularly legacy DoD systems). Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 10:41 AM To: m3devel Subject: [M3devel] Target.EOL Importance: Low I strongly suspect that Target.EOL can go away and just use Wr.EOL instead. Or even just \n. Cross builds are fairly rare, esp. cross between NT and Posix, and very many tools treat \n, \r\n, and perhaps \r the same, so even crossing NT <=> Posix doesn't likely matter. ie. it works, assuming you have a cross m3cg (ie: mingwin/cygwin hosted for NT hosts) gcc, Visual C++ compiler, m3front, all treat \r and \r\n the same. (witness all our *.h, *.c, *.i3, *.m3 files use \r\n) I think we have some tools that don't understand \r\n, but in my opinion that is a bug. All three formats should be treated the same. I know notepad doesn't display \n well, and Sun cc might not like \r\n. I know some compiler I used recently was picky, but "many" compilers are not. cmd might be ok with \n. Python calls it "universal newlines", treating all three formats the same. C:\dev2\cm3.2\m3-sys\cm3\src\Utils.m3(190):PROCEDURE CopyText (old, new: TEXT) = C:\dev2\cm3.2\m3-sys\m3middle\src\M3File.m3(61):PROCEDURE CopyText (src, dest: TEXT; eol: TEXT) RAISES {OSError.E} = probably leave alone, except that they don't treat \r correctly if you consider the historical Apple behavior. There are three formats, not two. Anyway, it's not a big deal. I think my problem here is solved by the change to m3x86.m3 I already made. I'll put Target.EOL back on my machine. - Jay ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 12 01:25:15 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 00:25:15 +0000 Subject: [M3devel] Target.EOL In-Reply-To: References: , Message-ID: Serial I/O is used plenty, for debugging. Though the protocols are generally "binary" not "text". Even any world with multiple systems messes with historical notions that one system is one way, other systems are another, and that they rarely see each other or exchange files. File exchange is very common these days, so it behooves every piece of new code to understand either format and possibly detect what the old code on the other side wants and be willing to write any of the three formats. "Constants" end up not useful. This isn't the EOL you care about. Target.EOL is written to .m3ship, foo.molog. It is used in m3front, m3middle, m3back, cm3 packages. It is exposed from m3middle/Target.i3. It is used by the old bootstrapping code, which is strewn around cm3, which I do use some of as part of my automation, uses it. (I don't use e.g. the makefile fragments that it outputs, but I use the behavior that stops after producing assembly, doesn't run the assembler or linker or C compiler) For example if you cross from NT386 to Posix, any \r\n in .man, .c, .h files will be changed to \n, and copied to the target directory. The thing is though, we don't have any \r\n anyway so that is a no-op. If you cross from Posix to NT386, \n will changed to \r\n. But it doesn't matter. Native builds just leave the \n alone -- which is what we always have -- and they work fine. If you cross from a hypothetical old Macintosh with its \r format, the newlines get completely destroyed because the code is wrong. Oops. Target.EOL can be changed to Wr.EOL or a hardcoded \n and almost nothing would change. The difference is that if we hardcoded \n, native NT386 users who happened to look at .m3ship with notepad, wouldn't have a good experience. Almost nobody ever looks at those files. Use Wr.EOL and then the only downside would be .m3ship files when crossing to NT386. And even still, .m3ship files don't participate in cross builds as I do them. My cross build only produces cm3, and then I build the rest native. Though there is a place for this perhaps. A cross build that builds everything, leaving final assembly, C compilation, linking and shipping to the target. We should consider that as a future distribution format. Similar to today's packages, but cross. But still, Wr.EOL will be fine. I'm very tempted to say the same thing about / and \, except that I recently experienced the pain of using an older NT386 cm3 that doesn't accept \. It behooves me to use \ "where appropriate" so I can be compatible with that. I need to fix scripts/python and/or cminstall/src/config-no-install a little. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Thu, 11 Mar 2010 17:03:30 -0500 Subject: RE: [M3devel] Target.EOL I don?t think I?ve used Target.EOL, but I?ve definitely used some EOL definition somewhere, maybe it was Wr.EOL, I?d have to check to be sure. I?m not sure the implications of removing Target.EOL, but whatever is done, we need to maintain a way to distinguish at runtime the proper line ending format for the host environment. I?ve written a lot of code in the past where this is important. Some of my code interfaces with other systems whose line ending formats have to be compared and/or reconciled with the host when doing serial data transfer (yes there are still systems that rely on good ?ol serial I/O, particularly legacy DoD systems). Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 10:41 AM To: m3devel Subject: [M3devel] Target.EOL Importance: Low I strongly suspect that Target.EOL can go away and just use Wr.EOL instead. Or even just \n. Cross builds are fairly rare, esp. cross between NT and Posix, and very many tools treat \n, \r\n, and perhaps \r the same, so even crossing NT <=> Posix doesn't likely matter. ie. it works, assuming you have a cross m3cg (ie: mingwin/cygwin hosted for NT hosts) gcc, Visual C++ compiler, m3front, all treat \r and \r\n the same. (witness all our *.h, *.c, *.i3, *.m3 files use \r\n) I think we have some tools that don't understand \r\n, but in my opinion that is a bug. All three formats should be treated the same. I know notepad doesn't display \n well, and Sun cc might not like \r\n. I know some compiler I used recently was picky, but "many" compilers are not. cmd might be ok with \n. Python calls it "universal newlines", treating all three formats the same. C:\dev2\cm3.2\m3-sys\cm3\src\Utils.m3(190):PROCEDURE CopyText (old, new: TEXT) = C:\dev2\cm3.2\m3-sys\m3middle\src\M3File.m3(61):PROCEDURE CopyText (src, dest: TEXT; eol: TEXT) RAISES {OSError.E} = probably leave alone, except that they don't treat \r correctly if you consider the historical Apple behavior. There are three formats, not two. Anyway, it's not a big deal. I think my problem here is solved by the change to m3x86.m3 I already made. I'll put Target.EOL back on my machine. - Jay CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Highjinks at gmx.com Fri Mar 12 01:35:25 2010 From: Highjinks at gmx.com (Chris) Date: Fri, 12 Mar 2010 01:35:25 +0100 (CET) Subject: [M3devel] Constant to Constant? Message-ID: <20100311203930.9b7cb406.Highjinks@gmx.com> Things are progressing here, slowly but steadily. I'm curious... When writing an interface for something like this ... /* C Type */ const C_Foo_t *Bar = get_foo_data(misc); Is it better to do it this way... <* EXTERNAL get_foo_data:C *> TYPE Foo_Data = PROCEDURE(get_foo_data(somemisc : misctype) : C_Foo_t; Or should I just do... UNSAFE INTERFACE Foo; (* Translating the C typedef struct over to Modula3 code. *) TYPE C_Foo_T = UNTRACED REF RECORD .... END; <* EXTERNAL get_foo_data:C *> PROCEDURE get_foo_data(somemisc : misctype) : C_foo_t; END Foo. Main.m3 IMPORT Foo; Foo_Data := Foo.get_foo_data(Misc); END Main. C_Foo_t is known, and has a Modula 3 representation in the Interface. But as the C Code shows, it has to be a constant value. What's the best way to do this without hosing the Interface for everyone else that might want to use it? Variable expressions are no problem, it's making it a constant that's giving me trouble. It's constant because the data structure is managed by the supporting library, not by the Modula3 Code. Tips...pointers? -- Chris From jay.krell at cornell.edu Fri Mar 12 02:06:57 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 01:06:57 +0000 Subject: [M3devel] Constant to Constant? In-Reply-To: <20100311203930.9b7cb406.Highjinks@gmx.com> References: <20100311203930.9b7cb406.Highjinks@gmx.com> Message-ID: Chris, I'm not sure I understand. Is the result "in misc", or is it constant relative to the program's lifetime? Or is it initialized on-demand and then constant from then on? There are a few instances where "constants" are initialized in Modula-3 module initializers. They are "VAR" from a technical point of view, in the .i3 file, but then VAR is immediately preceded by a comment that they are CONST. I do something similar in m3core/unix, except the data is actually const/static initialized in C code. I do that to avoid duplicating header content. Header content can be duplicated, if it is fairly portable and stable, only needs to be duplicated once, easy to get correct, and stay correct. The particular problem in m3core/unix is portability -- the header content would have to be duplicated differently for every target. Incorrectly duplicated C headers silently violate safety (see below). You should try to provide a SAFE interface, so that client code can be SAFE. And then, you are responsible for upholding what that means. Your module implementation might be UNSAFE. But clients must be protected from a "certain class of bug". For example, a claimed-to-be SAFE interface cannot expose stdlib/free() because a client could double free. This is an easy thing to mess up and it is very unfortunate when it is. It is roughly equivalent to bugs in the compiler or garbage collector. - Jay > From: Highjinks at gmx.com > To: m3devel at elegosoft.com > Date: Fri, 12 Mar 2010 01:35:25 +0100 > Subject: [M3devel] Constant to Constant? > > > Things are progressing here, slowly but steadily. > > I'm curious... > > When writing an interface for something like this ... > > /* C Type */ > const C_Foo_t *Bar = get_foo_data(misc); > > Is it better to do it this way... > > <* EXTERNAL get_foo_data:C *> > TYPE Foo_Data = PROCEDURE(get_foo_data(somemisc : misctype) : C_Foo_t; > > Or should I just do... > UNSAFE INTERFACE Foo; > > (* Translating the C typedef struct over to Modula3 code. *) > TYPE C_Foo_T = UNTRACED REF RECORD .... END; > > <* EXTERNAL get_foo_data:C *> > PROCEDURE get_foo_data(somemisc : misctype) : C_foo_t; > END Foo. > > Main.m3 > IMPORT Foo; > Foo_Data := Foo.get_foo_data(Misc); > END Main. > > C_Foo_t is known, and has a Modula 3 representation in the Interface. But as the C Code shows, it has to be a constant value. > > What's the best way to do this without hosing the Interface for everyone else that might want to use it? Variable expressions are no problem, it's making it a constant that's giving me trouble. It's constant because the data structure is managed by the supporting library, not by the Modula3 Code. > > Tips...pointers? > > -- > Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Mar 12 02:14:12 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 20:14:12 -0500 Subject: [M3devel] Ticket #1082 Message-ID: I have updated the comments on Ticket #1082 There are two postings, as shown below. Note that it looks like there is a problem with "START /WAIT ..." command differences between Windows versions, so we may need to open up a new Trac ticket for this issue (see comment #2 below). FIRST COMMENT: --------------------- The "startReactor.CMD" file is obsolete and no longer in the repository. The new files are in "scripts\install\windows" and should be copied to the "cm3\bin" folder on a windows installation. "cm3StartIDE.CMD" replaces "startReactor.CMD". This file makes use of "cm3CommandShell.CMD", a script that ensures the environment is set up properly, including Visual C++. It appears that the problem reporter has his installation rooted at E:\cm3 rather than C:\cm3. So, he will need to set an environment variable "set CM3_ROOT=E:\cm3" to let these scripts know where to find the cm3 installation. Alternately, cm3CommandShell can be invoked with the arguments "Root E:\cm3" to tell it where to find the installation. Today, I checked in an update to cm3CommandShell.CMD that allows it to search the current PATH environment variable in an attempt to find the root of the cm3 installation, so now if "E:\cm3\bin" is in your path, it will find it without having to set the CM3_ROOT or invoking with "Root E:\cm3". Hope this helps! Note also that strictly speaking, these CMD files are not necessary; rather they are conveniences I've found useful under Windows. I also can supply some .REG files that will allow you to integrate these into the Explorer shell so that you can start a cm3 command shell in any folder and you can start CM3IDE. SECOND COMMENT: --------------------- Ok as for CM3IDE timing out, I've been able to reproduce this problem on Vista. There seems to be some difference I don't understand yet between the way the various Windows versions deal with the "START /WAIT ..." command. Here is a "quick fix" until we come up with a better solution. Note that the downside here is that when you terminate your browser, CM3IDE will stay running, instead of terminating. You can restart your browser and point it back to the http://localhost:3800/ URL and verify that it reconnects to CM3IDE. So, to terminate CM3IDE, you will have to use CTRL-C from its console output window. Now for the quick fix: 1. Go to your CM3_IDE_HOME folder (e.g., C:\Users\RColeburn\Documents\CM3_IDE_Home). 2. Open the most recent CM3_IDE.cfg file in notepad (e.g., notepad CM3_IDE.cfg1). 3. Edit the line toward the end of the file that begins with "proc start_browser". On this line change TRUE to FALSE. Note that case is significant. Now try cm3IDE.exe, or cm3_StartIDE.CMD. It should work for you now as described above. Regards, Randy Coleburn ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Mar 12 02:38:34 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 20:38:34 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait Message-ID: Ok, I've done a bit more testing. I don't think the problem has to do with "START /WAIT ..." but rather it seems to be a difference in Windows Internet Explorer. When your browser is set to FIREFOX instead of IE, the "START /WAIT firefox.exe" command works fine, so you can keep the "return TRUE" in the CM3_IDE.cfg file. But, when using IE8, the "START /WAIT iexplorer.exe" command does not wait for IE8 to terminate and returns immediately, thus the "return TRUE" in CM3_IDE.cfg must be changed to "return FALSE" to prevent immediate shutdown of CM3IDE. If I go to a Windows 2000 box running IE6, the "START /WAIT" again seems to wait for IE to terminate. I haven't tested yet with IE7 to see if it behaves correctly or not. So for now, folks may want to use FireFox instead of IE with CM3IDE. Regards, Randy ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 12 02:43:03 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 01:43:03 +0000 Subject: [M3devel] more info on ticket 1082 wrt start /wait In-Reply-To: References: Message-ID: What if you just simply: "start http://localhost:whateverport" Personally I find cm3ide/reactor to be ..not great. The behavior you are noticing is that the web browser might decide there is another adequate instance of it running and ask it to display something and exit. You will likely find different behavior depending on if an instance of the browser is already running. There might also be a command line option to control this. But I think your best best is just not to use /wait. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Thu, 11 Mar 2010 20:38:34 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait Ok, I?ve done a bit more testing. I don?t think the problem has to do with ?START /WAIT ?? but rather it seems to be a difference in Windows Internet Explorer. When your browser is set to FIREFOX instead of IE, the ?START /WAIT firefox.exe? command works fine, so you can keep the ?return TRUE? in the CM3_IDE.cfg file. But, when using IE8, the ?START /WAIT iexplorer.exe? command does not wait for IE8 to terminate and returns immediately, thus the ?return TRUE? in CM3_IDE.cfg must be changed to ?return FALSE? to prevent immediate shutdown of CM3IDE. If I go to a Windows 2000 box running IE6, the ?START /WAIT? again seems to wait for IE to terminate. I haven?t tested yet with IE7 to see if it behaves correctly or not. So for now, folks may want to use FireFox instead of IE with CM3IDE. Regards, Randy CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 12 02:36:53 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 01:36:53 +0000 Subject: [M3devel] Ticket #1082 In-Reply-To: References: Message-ID: > There seems to be some difference I don't understand yet between > the way the various Windows versions deal with the "START /WAIT ..." command. Please elaborate. What varying behaviors are you seeing on different versions? I have used start /wait a fair number of times and never noticed any differences among Windows versions, at least on NT versions since circa NT 4. Definitely maybe some versions don't have any sort of start nor /wait, and start on Win9x is probably different than NT. start is a builtin command to NT cmd but an .exe on Win9x, as I vaguely recall. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Thu, 11 Mar 2010 20:14:12 -0500 Subject: [M3devel] Ticket #1082 I have updated the comments on Ticket #1082 There are two postings, as shown below. Note that it looks like there is a problem with ?START /WAIT ?? command differences between Windows versions, so we may need to open up a new Trac ticket for this issue (see comment #2 below). FIRST COMMENT: --------------------- The "startReactor.CMD" file is obsolete and no longer in the repository. The new files are in "scripts\install\windows" and should be copied to the "cm3\bin" folder on a windows installation. "cm3StartIDE.CMD" replaces "startReactor.CMD". This file makes use of "cm3CommandShell.CMD", a script that ensures the environment is set up properly, including Visual C++. It appears that the problem reporter has his installation rooted at E:\cm3 rather than C:\cm3. So, he will need to set an environment variable "set CM3_ROOT=E:\cm3" to let these scripts know where to find the cm3 installation. Alternately, cm3CommandShell can be invoked with the arguments "Root E:\cm3" to tell it where to find the installation. Today, I checked in an update to cm3CommandShell.CMD that allows it to search the current PATH environment variable in an attempt to find the root of the cm3 installation, so now if "E:\cm3\bin" is in your path, it will find it without having to set the CM3_ROOT or invoking with "Root E:\cm3". Hope this helps! Note also that strictly speaking, these CMD files are not necessary; rather they are conveniences I?ve found useful under Windows. I also can supply some .REG files that will allow you to integrate these into the Explorer shell so that you can start a cm3 command shell in any folder and you can start CM3IDE. SECOND COMMENT: --------------------- Ok as for CM3IDE timing out, I've been able to reproduce this problem on Vista. There seems to be some difference I don't understand yet between the way the various Windows versions deal with the "START /WAIT ..." command. Here is a "quick fix" until we come up with a better solution. Note that the downside here is that when you terminate your browser, CM3IDE will stay running, instead of terminating. You can restart your browser and point it back to the http://localhost:3800/ URL and verify that it reconnects to CM3IDE. So, to terminate CM3IDE, you will have to use CTRL-C from its console output window. Now for the quick fix: 1. Go to your CM3_IDE_HOME folder (e.g., C:\Users\RColeburn\Documents\CM3_IDE_Home). 2. Open the most recent CM3_IDE.cfg file in notepad (e.g., notepad CM3_IDE.cfg1). 3. Edit the line toward the end of the file that begins with "proc start_browser". On this line change TRUE to FALSE. Note that case is significant. Now try cm3IDE.exe, or cm3_StartIDE.CMD. It should work for you now as described above. Regards, Randy Coleburn CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Mar 12 02:46:13 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 20:46:13 -0500 Subject: [M3devel] Constant to Constant? In-Reply-To: <20100311203930.9b7cb406.Highjinks@gmx.com> References: <20100311203930.9b7cb406.Highjinks@gmx.com> Message-ID: Chris: You may also want to look at pkg\m3core\src\win32\WinUser.i3 Regards, Randy -----Original Message----- From: Chris [mailto:Highjinks at gmx.com] Sent: Thursday, March 11, 2010 7:35 PM To: m3devel at elegosoft.com Subject: [M3devel] Constant to Constant? Things are progressing here, slowly but steadily. I'm curious... When writing an interface for something like this ... /* C Type */ const C_Foo_t *Bar = get_foo_data(misc); Is it better to do it this way... <* EXTERNAL get_foo_data:C *> TYPE Foo_Data = PROCEDURE(get_foo_data(somemisc : misctype) : C_Foo_t; Or should I just do... UNSAFE INTERFACE Foo; (* Translating the C typedef struct over to Modula3 code. *) TYPE C_Foo_T = UNTRACED REF RECORD .... END; <* EXTERNAL get_foo_data:C *> PROCEDURE get_foo_data(somemisc : misctype) : C_foo_t; END Foo. Main.m3 IMPORT Foo; Foo_Data := Foo.get_foo_data(Misc); END Main. C_Foo_t is known, and has a Modula 3 representation in the Interface. But as the C Code shows, it has to be a constant value. What's the best way to do this without hosing the Interface for everyone else that might want to use it? Variable expressions are no problem, it's making it a constant that's giving me trouble. It's constant because the data structure is managed by the supporting library, not by the Modula3 Code. Tips...pointers? -- Chris CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. From rcolebur at SCIRES.COM Fri Mar 12 03:06:36 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 21:06:36 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait In-Reply-To: References: Message-ID: Jay et al: Well I've done more testing. Jay you are right about the multiple instances. For both IE8 and FireFox on Vista, if you already have an instance running and use "START /WAIT" to get another instance, the second instance starts up (a new window) and the "START /WAIT" immediately returns rather than waiting. Conversely, if this is the first instance of IE8 or FireFox, then "START /WAIT" does indeed wait for the browser to close before it returns. I'll have to get to an XP machine to confirm if this same behavior occurs there as well. If you go back to IE6, it didn't have Tabbed windows, so maybe that is why "START /WAIT" always waited. I don't have an immediate fix for this behavior since the original design used the "START /WAIT" functionality on Windows. On POSIX, it doesn't. The quick fix I mentioned earlier is probably the best thing to do now. I could perhaps modify the sources to change the default CFG to "return FALSE" on Windows instead of "return TRUE", thereby implementing the quick fix for everyone. I'll check into this shortly. As for your comments about cm3ide not being "great" that is your opinion; however, back in the day when reactor (aka cm3ide) first came out, this was indeed leading edge stuff. The idea that you could tweak your IDE by making simple HTML file changes I think was pretty novel at the time, and the thinking from the CMass Inc folks was that as the browser improved, your IDE would improve automatically (e.g. multiple tabs was a subsequent browser improvement). Even though you may not consider CM3IDE to be a great IDE, I do find that it is extremely helpful during development when browsing source code and looking up interface specs. With just a hyperlink, you can quickly find what you are looking for. It also has a lot of built-in links to reference material and coding examples. Note that cm3ide hasn't really changed much at all since reactor first debuted. Now that we finally have the open-source release, we can modify it to make it better. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 8:43 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait What if you just simply: "start http://localhost:whateverport" Personally I find cm3ide/reactor to be ..not great. The behavior you are noticing is that the web browser might decide there is another adequate instance of it running and ask it to display something and exit. You will likely find different behavior depending on if an instance of the browser is already running. There might also be a command line option to control this. But I think your best best is just not to use /wait. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Thu, 11 Mar 2010 20:38:34 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait Ok, I've done a bit more testing. I don't think the problem has to do with "START /WAIT ..." but rather it seems to be a difference in Windows Internet Explorer. When your browser is set to FIREFOX instead of IE, the "START /WAIT firefox.exe" command works fine, so you can keep the "return TRUE" in the CM3_IDE.cfg file. But, when using IE8, the "START /WAIT iexplorer.exe" command does not wait for IE8 to terminate and returns immediately, thus the "return TRUE" in CM3_IDE.cfg must be changed to "return FALSE" to prevent immediate shutdown of CM3IDE. If I go to a Windows 2000 box running IE6, the "START /WAIT" again seems to wait for IE to terminate. I haven't tested yet with IE7 to see if it behaves correctly or not. So for now, folks may want to use FireFox instead of IE with CM3IDE. Regards, Randy ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 12 03:47:53 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 11 Mar 2010 21:47:53 -0500 Subject: [M3devel] Target.EOL In-Reply-To: References: Message-ID: <5F2763E1-801E-4EB2-BFAC-884EB53DC3C6@cs.purdue.edu> On 11 Mar 2010, at 17:03, Coleburn, Randy wrote: > I don?t think I?ve used Target.EOL, but I?ve definitely used some EOL definition somewhere, maybe it was Wr.EOL, I?d have to check to be sure. This is part of the compiler subsystem and its tool interfaces. I don't think it will affect application code. If we are to default I suggest we go with the more compact \n termination instead of verbose \r\n. > > I?m not sure the implications of removing Target.EOL, but whatever is done, we need to maintain a way to distinguish at runtime the proper line ending format for the host environment. > > I?ve written a lot of code in the past where this is important. Some of my code interfaces with other systems whose line ending formats have to be compared and/or reconciled with the host when doing serial data transfer (yes there are still systems that rely on good ?ol serial I/O, particularly legacy DoD systems). > > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Thursday, March 11, 2010 10:41 AM > To: m3devel > Subject: [M3devel] Target.EOL > Importance: Low > > I strongly suspect that Target.EOL can go away and just use Wr.EOL instead. > Or even just \n. > > > Cross builds are fairly rare, esp. cross between NT and Posix, and very > many tools treat \n, \r\n, and perhaps \r the same, so even > crossing NT <=> Posix doesn't likely matter. > ie. it works, assuming you have a cross m3cg (ie: mingwin/cygwin hosted for NT hosts) > > > gcc, Visual C++ compiler, m3front, all treat \r and \r\n the same. > (witness all our *.h, *.c, *.i3, *.m3 files use \r\n) > I think we have some tools that don't understand \r\n, but in my opinion that is a bug. > All three formats should be treated the same. > > > I know notepad doesn't display \n well, and Sun cc might not like \r\n. > I know some compiler I used recently was picky, but "many" compilers are not. > cmd might be ok with \n. > Python calls it "universal newlines", treating all three formats the same. > > > C:\dev2\cm3.2\m3-sys\cm3\src\Utils.m3(190):PROCEDURE CopyText (old, new: TEXT) = > C:\dev2\cm3.2\m3-sys\m3middle\src\M3File.m3(61):PROCEDURE CopyText (src, dest: TEXT; eol: TEXT) RAISES {OSError.E} = > > > probably leave alone, except that they don't treat \r correctly if you consider the historical Apple behavior. > There are three formats, not two. > > > Anyway, it's not a big deal. I think my problem here is solved by the change to m3x86.m3 I already made. > I'll put Target.EOL back on my machine. > > > - Jay > > CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. > > EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 12 03:49:39 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 02:49:39 +0000 Subject: [M3devel] more info on ticket 1082 wrt start /wait In-Reply-To: References: , , Message-ID: Start /wait is always doing the same thing, it is always waiting. But what it is launching is deciding to exit right away, or not. More later.. From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Thu, 11 Mar 2010 21:06:36 -0500 Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait Jay et al: Well I?ve done more testing. Jay you are right about the multiple instances. For both IE8 and FireFox on Vista, if you already have an instance running and use ?START /WAIT? to get another instance, the second instance starts up (a new window) and the ?START /WAIT? immediately returns rather than waiting. Conversely, if this is the first instance of IE8 or FireFox, then ?START /WAIT? does indeed wait for the browser to close before it returns. I?ll have to get to an XP machine to confirm if this same behavior occurs there as well. If you go back to IE6, it didn?t have Tabbed windows, so maybe that is why ?START /WAIT? always waited. I don?t have an immediate fix for this behavior since the original design used the ?START /WAIT? functionality on Windows. On POSIX, it doesn?t. The quick fix I mentioned earlier is probably the best thing to do now. I could perhaps modify the sources to change the default CFG to ?return FALSE? on Windows instead of ?return TRUE?, thereby implementing the quick fix for everyone. I?ll check into this shortly. As for your comments about cm3ide not being ?great? that is your opinion; however, back in the day when reactor (aka cm3ide) first came out, this was indeed leading edge stuff. The idea that you could tweak your IDE by making simple HTML file changes I think was pretty novel at the time, and the thinking from the CMass Inc folks was that as the browser improved, your IDE would improve automatically (e.g. multiple tabs was a subsequent browser improvement). Even though you may not consider CM3IDE to be a great IDE, I do find that it is extremely helpful during development when browsing source code and looking up interface specs. With just a hyperlink, you can quickly find what you are looking for. It also has a lot of built-in links to reference material and coding examples. Note that cm3ide hasn?t really changed much at all since reactor first debuted. Now that we finally have the open-source release, we can modify it to make it better. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 8:43 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait What if you just simply: "start http://localhost:whateverport" Personally I find cm3ide/reactor to be ..not great. The behavior you are noticing is that the web browser might decide there is another adequate instance of it running and ask it to display something and exit. You will likely find different behavior depending on if an instance of the browser is already running. There might also be a command line option to control this. But I think your best best is just not to use /wait. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Thu, 11 Mar 2010 20:38:34 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait Ok, I?ve done a bit more testing. I don?t think the problem has to do with ?START /WAIT ?? but rather it seems to be a difference in Windows Internet Explorer. When your browser is set to FIREFOX instead of IE, the ?START /WAIT firefox.exe? command works fine, so you can keep the ?return TRUE? in the CM3_IDE.cfg file. But, when using IE8, the ?START /WAIT iexplorer.exe? command does not wait for IE8 to terminate and returns immediately, thus the ?return TRUE? in CM3_IDE.cfg must be changed to ?return FALSE? to prevent immediate shutdown of CM3IDE. If I go to a Windows 2000 box running IE6, the ?START /WAIT? again seems to wait for IE to terminate. I haven?t tested yet with IE7 to see if it behaves correctly or not. So for now, folks may want to use FireFox instead of IE with CM3IDE. Regards, Randy CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Mar 12 04:37:48 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 22:37:48 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait In-Reply-To: References: , , Message-ID: Jay: Yes, you said it better than I did. The problem is with the browser behavior, not with START /WAIT. I've implemented a quick-fix in the source code to change the default behavior on Windows to "return FALSE" on the "start_browser" function instead of "return TRUE". That way, regardless of whether or not the browser exits right away, CM3IDE will stay running until it is manually terminated via CTRL-C in its console output window. (Note: "return TRUE" means that CM3IDE will terminate when the browser terminates, whereas "return FALSE" means that CM3IDE will stay running even after the browser terminates. On a server system where you have multiple network clients connecting to the single CM3IDE instance running on the server, you would always want to use "return FALSE". Likewise, on a Unix system where you start CM3IDE as a background process when the system boots, you would want to use "return FALSE".) I am already thinking of alternate solutions and an easy way for CM3IDE to limit itself to a single instance. I'll work on polishing these up in the next few days, but the quick-fix should solve the immediate problem for now. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 9:50 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait Start /wait is always doing the same thing, it is always waiting. But what it is launching is deciding to exit right away, or not. More later.. ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Thu, 11 Mar 2010 21:06:36 -0500 Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait Jay et al: Well I've done more testing. Jay you are right about the multiple instances. For both IE8 and FireFox on Vista, if you already have an instance running and use "START /WAIT" to get another instance, the second instance starts up (a new window) and the "START /WAIT" immediately returns rather than waiting. Conversely, if this is the first instance of IE8 or FireFox, then "START /WAIT" does indeed wait for the browser to close before it returns. I'll have to get to an XP machine to confirm if this same behavior occurs there as well. If you go back to IE6, it didn't have Tabbed windows, so maybe that is why "START /WAIT" always waited. I don't have an immediate fix for this behavior since the original design used the "START /WAIT" functionality on Windows. On POSIX, it doesn't. The quick fix I mentioned earlier is probably the best thing to do now. I could perhaps modify the sources to change the default CFG to "return FALSE" on Windows instead of "return TRUE", thereby implementing the quick fix for everyone. I'll check into this shortly. As for your comments about cm3ide not being "great" that is your opinion; however, back in the day when reactor (aka cm3ide) first came out, this was indeed leading edge stuff. The idea that you could tweak your IDE by making simple HTML file changes I think was pretty novel at the time, and the thinking from the CMass Inc folks was that as the browser improved, your IDE would improve automatically (e.g. multiple tabs was a subsequent browser improvement). Even though you may not consider CM3IDE to be a great IDE, I do find that it is extremely helpful during development when browsing source code and looking up interface specs. With just a hyperlink, you can quickly find what you are looking for. It also has a lot of built-in links to reference material and coding examples. Note that cm3ide hasn't really changed much at all since reactor first debuted. Now that we finally have the open-source release, we can modify it to make it better. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 8:43 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait What if you just simply: "start http://localhost:whateverport" Personally I find cm3ide/reactor to be ..not great. The behavior you are noticing is that the web browser might decide there is another adequate instance of it running and ask it to display something and exit. You will likely find different behavior depending on if an instance of the browser is already running. There might also be a command line option to control this. But I think your best best is just not to use /wait. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Thu, 11 Mar 2010 20:38:34 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait Ok, I've done a bit more testing. I don't think the problem has to do with "START /WAIT ..." but rather it seems to be a difference in Windows Internet Explorer. When your browser is set to FIREFOX instead of IE, the "START /WAIT firefox.exe" command works fine, so you can keep the "return TRUE" in the CM3_IDE.cfg file. But, when using IE8, the "START /WAIT iexplorer.exe" command does not wait for IE8 to terminate and returns immediately, thus the "return TRUE" in CM3_IDE.cfg must be changed to "return FALSE" to prevent immediate shutdown of CM3IDE. If I go to a Windows 2000 box running IE6, the "START /WAIT" again seems to wait for IE to terminate. I haven't tested yet with IE7 to see if it behaves correctly or not. So for now, folks may want to use FireFox instead of IE with CM3IDE. Regards, Randy ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 12 07:45:07 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 06:45:07 +0000 Subject: [M3devel] merits/demerits of cm3ide In-Reply-To: References: , , Message-ID: > cm3ide > more later Being at the mercy of the rapidly changing web browsers is a very sharp double edged sword. An IDE with no editor and no debugger..is not an IDE. Maybe it is a source browser. Maybe an IDE is not worth it. People are very slow to change editors. Debuggers are hard to write. Probably better to try integrating with an actual IDE like VisualStudio or Eclipse. Or to have a decent enough language, library, build system, that the language doesn't need such costly support. Maybe we do have such a think. Hyperlinking the source is nice, but all I need is "find in files". I can open the documentation from a file system browser, which open it in a web browser. I should just be able to use online documentation anyway, not local stuff. Having the real compiler output enough information such that writing other lanugage-aware tools, such as hyperlinked source generation, is also good. Too many systems have their IDE write another parser, and no two parsers agree. However it is a tough problem, because the compiler's job is different than a source browser. For example, a compiler must reject incorrect source, but a source browser should be lenient. Relying on the browser is a cheap way to get a somewhat decent somewhat portable very limited gui. That's about it. I don't think it is fixable. It's just pretty much all wierd and wrong. I was quite shocked the first time I used it, having heard it described as an IDE. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Mar 12 10:20:59 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Fri, 12 Mar 2010 04:20:59 -0500 Subject: [M3devel] merits/demerits of cm3ide In-Reply-To: References: , , Message-ID: Jay as far as editor goes, CM3IDE lets you use the editor of your choice. Once you define it, CM3IDE calls upon your defined editor when you want to edit a source file. Note also that when you compile and get errors, CM3IDE annotates the listing and gives hyperlinks to allow you to make edits. When you click on a line with a syntax error and choose to edit it, CM3IDE calls upon your editor and tells it to position the cursor at that line so you can make the fix easily. CM3IDE also shows for each interface all modules that import that interface. This is useful also when you are making an interface change that will break modules, you can quickly find all the ones affected and make the edits. Granted you can use command line tools for finding stuff in files, but I personally like the hyper linking. In my environment, I use CM3IDE, the CRiSP programmers editor, and TortoiseCVS. CRiSP integrates with CVS so you can check files in/out of the repository automatically, all controlled from CM3IDE. There is no point in arguing further about merits/demerits of CM3IDE. I didn't write it; I simply worked to make it open-source. If you don't like it, don't use it. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, March 12, 2010 1:45 AM To: Coleburn, Randy; m3devel Subject: merits/demerits of cm3ide > cm3ide > more later Being at the mercy of the rapidly changing web browsers is a very sharp double edged sword. An IDE with no editor and no debugger..is not an IDE. Maybe it is a source browser. Maybe an IDE is not worth it. People are very slow to change editors. Debuggers are hard to write. Probably better to try integrating with an actual IDE like VisualStudio or Eclipse. Or to have a decent enough language, library, build system, that the language doesn't need such costly support. Maybe we do have such a think. Hyperlinking the source is nice, but all I need is "find in files". I can open the documentation from a file system browser, which open it in a web browser. I should just be able to use online documentation anyway, not local stuff. Having the real compiler output enough information such that writing other lanugage-aware tools, such as hyperlinked source generation, is also good. Too many systems have their IDE write another parser, and no two parsers agree. However it is a tough problem, because the compiler's job is different than a source browser. For example, a compiler must reject incorrect source, but a source browser should be lenient. Relying on the browser is a cheap way to get a somewhat decent somewhat portable very limited gui. That's about it. I don't think it is fixable. It's just pretty much all wierd and wrong. I was quite shocked the first time I used it, having heard it described as an IDE. - Jay ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Fri Mar 12 15:17:34 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 12 Mar 2010 15:17:34 +0100 Subject: [M3devel] merits/demerits of cm3ide In-Reply-To: References: , , Message-ID: <20100312151734.r2i0ui5nkg0k0woc@mail.elegosoft.com> Well, some people have used Reactor/cm3ide and liked it. There have been several requests for it in recent years. It's certainly no match for a large IDE framework like Eclipse, but it's the only GUI for the compiler we have. As for it not being an IDE, I do not really agree; it should already integrate compiling, shipping, source viewing, searching, pretty printing. And it integrates on the level of language elements likes types, interfaces etc. We should also be able to integrate a debugger (at least m3gdb); this has been done to gdb with other GUIs, too. I'd also like to see an M3 plugin for Eclipse, but nobody seems to be working on that. I myself am still happy with my XEmacs editor whenever I actually get to browse or write code ;-) So just view it as an offer and keep not using it if you don't like it. There are so many things that need work in the cm3 system that work won't get short during the next years. Olaf Quoting Jay K : > > cm3ide > > more later > > Being at the mercy of the rapidly changing web browsers is a very > sharp double edged sword. > > An IDE with no editor and no debugger..is not an IDE. > Maybe it is a source browser. > > Maybe an IDE is not worth it. > > People are very slow to change editors. > > Debuggers are hard to write. > > Probably better to try integrating with an actual IDE like > VisualStudio or Eclipse. > > Or to have a decent enough language, library, build system, that > the language doesn't > need such costly support. Maybe we do have such a think. > > Hyperlinking the source is nice, but all I need is "find in files". > I can open the documentation from a file system browser, which > open it in a web browser. I should just be able to use online > documentation anyway, not local stuff. > > Having the real compiler output enough information > such that writing other lanugage-aware tools, such > as hyperlinked source generation, is also good. > Too many systems have their IDE write another parser, > and no two parsers agree. However it is a tough problem, > because the compiler's job is different than a source > browser. For example, a compiler must reject incorrect > source, but a source browser should be lenient. > > Relying on the browser is a cheap way to get a somewhat > decent somewhat portable very limited gui. > That's about it. > > I don't think it is fixable. > It's just pretty much all wierd and wrong. > > I was quite shocked the first time I used it, having > heard it described as an IDE. -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From rcolebur at SCIRES.COM Sat Mar 13 06:57:00 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sat, 13 Mar 2010 00:57:00 -0500 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20100313011125.34CF12474001@birch.elegosoft.com> Message-ID: Jay: This is example code. The program terminates after the message box is displayed. There is no further processing. I did not write this example; I merely fixed it to compile again after someone changed the names of the procedures in the M3toC interface. Nevertheless, it is probably a bad "example" to foist on people by not also following the established pattern of freeing the storage , so I'll make a modification for this shortly. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, March 12, 2010 11:47 PM To: m3commit at elegosoft.com; Coleburn, Randy Subject: RE: [M3commit] CVS Update: cm3 This change is suspicious. I think it leaks. Please see http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/os/WIN32/FSWin32.m3?rev=1.7;content-type=text%2Fplain for the pattern that is usually used. Granted, given the conservativeness of the collector -- it finds stuff in registers and the stack without a need for metadata telling it where the points are, and the fact that we no longer have VM-synchronization, I don't understand the need for the usual pattern. Maybe because it is willing to move stuff? Therefore a non-traced copy is made for C that won't ever move? - Jay > Date: Sat, 13 Mar 2010 02:11:24 +0000 > To: m3commit at elegosoft.com > From: rcoleburn at elego.de > Subject: [M3commit] CVS Update: cm3 > > CVSROOT: /usr/cvs > Changes by: rcoleburn at birch. 10/03/13 02:11:24 > > Modified files: > cm3/examples/calling-c-win32/src/: OK.m3 > > Log message: > M3toC.TtoS has been renamed to M3toC.SharedTtoS, so make adjustments to Ok.m3 so that it will compile.--R.Coleburn > ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 13 07:36:25 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 13 Mar 2010 06:36:25 +0000 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20100313011125.34CF12474001@birch.elegosoft.com>, , Message-ID: Bad examples beget bad code. Bad (and good) examples get copied into real code. Don't make them. It wasn't a rename, it was I believe a semantic change accompanied by a rename. Better to not compile than compile and be incorrect. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Sat, 13 Mar 2010 00:57:00 -0500 Subject: Re: [M3devel] [M3commit] CVS Update: cm3 Jay: This is example code. The program terminates after the message box is displayed. There is no further processing. I did not write this example; I merely fixed it to compile again after someone changed the names of the procedures in the M3toC interface. Nevertheless, it is probably a bad ?example? to foist on people by not also following the established pattern of freeing the storage , so I?ll make a modification for this shortly. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, March 12, 2010 11:47 PM To: m3commit at elegosoft.com; Coleburn, Randy Subject: RE: [M3commit] CVS Update: cm3 This change is suspicious. I think it leaks. Please see http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/os/WIN32/FSWin32.m3?rev=1.7;content-type=text%2Fplain for the pattern that is usually used. Granted, given the conservativeness of the collector -- it finds stuff in registers and the stack without a need for metadata telling it where the points are, and the fact that we no longer have VM-synchronization, I don't understand the need for the usual pattern. Maybe because it is willing to move stuff? Therefore a non-traced copy is made for C that won't ever move? - Jay > Date: Sat, 13 Mar 2010 02:11:24 +0000 > To: m3commit at elegosoft.com > From: rcoleburn at elego.de > Subject: [M3commit] CVS Update: cm3 > > CVSROOT: /usr/cvs > Changes by: rcoleburn at birch. 10/03/13 02:11:24 > > Modified files: > cm3/examples/calling-c-win32/src/: OK.m3 > > Log message: > M3toC.TtoS has been renamed to M3toC.SharedTtoS, so make adjustments to Ok.m3 so that it will compile.--R.Coleburn > CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Sat Mar 13 07:49:55 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sat, 13 Mar 2010 01:49:55 -0500 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20100313011125.34CF12474001@birch.elegosoft.com>, , Message-ID: Jay: I agree "bad examples beget bad code", but I don't need to be lectured. I didn't create this example code. The example code is from the "Reactor v4.1 days and was produced by Critical Mass, Inc." Regardless of how the M3toC interface has changed since that time, it is obvious that the example code was not updated to be consistent with the changes. Thus, we had an example that failed to even compile, and that's not good either, especially since CM3IDE creates a copy of the example package in your private repository and lets you try to build and run it. I have repaired this example so that it compiles and runs. I've also made the change to free the memory to serve as a better exemplar of proper practice, even though it really doesn't affect the running of the program since the program terminates after the message box is closed and no further processing is done. I agree it was bad style in the first place on the part of the folks at Critical Mass. Now it is better. You will probably find other bad style examples, so if you want to fix them, be my guest. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Saturday, March 13, 2010 1:36 AM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] [M3commit] CVS Update: cm3 Bad examples beget bad code. Bad (and good) examples get copied into real code. Don't make them. It wasn't a rename, it was I believe a semantic change accompanied by a rename. Better to not compile than compile and be incorrect. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Sat, 13 Mar 2010 00:57:00 -0500 Subject: Re: [M3devel] [M3commit] CVS Update: cm3 Jay: This is example code. The program terminates after the message box is displayed. There is no further processing. I did not write this example; I merely fixed it to compile again after someone changed the names of the procedures in the M3toC interface. Nevertheless, it is probably a bad "example" to foist on people by not also following the established pattern of freeing the storage , so I'll make a modification for this shortly. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, March 12, 2010 11:47 PM To: m3commit at elegosoft.com; Coleburn, Randy Subject: RE: [M3commit] CVS Update: cm3 This change is suspicious. I think it leaks. Please see http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/os/WIN32/FSWin32.m3?rev=1.7;content-type=text%2Fplain for the pattern that is usually used. Granted, given the conservativeness of the collector -- it finds stuff in registers and the stack without a need for metadata telling it where the points are, and the fact that we no longer have VM-synchronization, I don't understand the need for the usual pattern. Maybe because it is willing to move stuff? Therefore a non-traced copy is made for C that won't ever move? - Jay > Date: Sat, 13 Mar 2010 02:11:24 +0000 > To: m3commit at elegosoft.com > From: rcoleburn at elego.de > Subject: [M3commit] CVS Update: cm3 > > CVSROOT: /usr/cvs > Changes by: rcoleburn at birch. 10/03/13 02:11:24 > > Modified files: > cm3/examples/calling-c-win32/src/: OK.m3 > > Log message: > M3toC.TtoS has been renamed to M3toC.SharedTtoS, so make adjustments to Ok.m3 so that it will compile.--R.Coleburn > ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 13 11:19:21 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 13 Mar 2010 10:19:21 +0000 Subject: [M3devel] comparisons vs. subranges Message-ID: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 13 11:49:37 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 13 Mar 2010 10:49:37 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: Message-ID: I might be able to do this. Give me a day or so.. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu; m3devel at elegosoft.com Date: Sat, 13 Mar 2010 10:19:21 +0000 Subject: [M3devel] comparisons vs. subranges <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Sat Mar 13 19:29:24 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Sat, 13 Mar 2010 13:29:24 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: Message-ID: <20100313182924.GA14329@topoi.pooq.com> On Sat, Mar 13, 2010 at 10:19:21AM +0000, Jay K wrote: > > <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; > <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; > > > > > Seems to me, the front end should notice these. > > The first should always be TRUE. > > And possibly, possibly warn. > > The second should always be FALSE. > > And possibly, possibly warn. > > > > "Generic" programming often hits this sort of thing though, a good reason not to warn. > > Programmer might also be working with existing code and have changed INTEGER to CARDINAL. > > Or be defending against future maintainers changing CARDINAL to INTEGER. Wasn't there a discussion a while ago about subranges out-of-bounds not being a safety problem? (Or was it arithmetic overflow?) This optimisation might well cause a quite hard-to-find bug if we don't guarantee subrange integrity. -- hendrik From jay.krell at cornell.edu Sat Mar 13 17:57:27 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 13 Mar 2010 16:57:27 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <20100313182924.GA14329@topoi.pooq.com> References: , <20100313182924.GA14329@topoi.pooq.com> Message-ID: Integer overflow is not a safety problem. That was the news (to me). Subranges do need to be enforced, at certain points, for their to be safety. This change doesn't change that. (Compiler bugs break safety, as always.) - Jay > Date: Sat, 13 Mar 2010 13:29:24 -0500 > From: hendrik at topoi.pooq.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] comparisons vs. subranges > > On Sat, Mar 13, 2010 at 10:19:21AM +0000, Jay K wrote: > > > > <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; > > <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; > > > > > > > > > > Seems to me, the front end should notice these. > > > > The first should always be TRUE. > > > > And possibly, possibly warn. > > > > The second should always be FALSE. > > > > And possibly, possibly warn. > > > > > > > > "Generic" programming often hits this sort of thing though, a good reason not to warn. > > > > Programmer might also be working with existing code and have changed INTEGER to CARDINAL. > > > > Or be defending against future maintainers changing CARDINAL to INTEGER. > > Wasn't there a discussion a while ago about subranges out-of-bounds not > being a safety problem? (Or was it arithmetic overflow?) This > optimisation might well cause a quite hard-to-find bug if we don't > guarantee subrange integrity. > > -- hendrik -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sat Mar 13 19:03:18 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sat, 13 Mar 2010 13:03:18 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: Message-ID: Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: > <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; > <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; > > > Seems to me, the front end should notice these. > The first should always be TRUE. > And possibly, possibly warn. > The second should always be FALSE. > And possibly, possibly warn. > > "Generic" programming often hits this sort of thing though, a good reason not to warn. > Programmer might also be working with existing code and have changed INTEGER to CARDINAL. > Or be defending against future maintainers changing CARDINAL to INTEGER. > > > The backend isn't give enough information, because CARDINAL = INTEGER as far > as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. > The only way to get unsigned types is to use ADDRESS from what I see. > > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 13 19:26:34 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 13 Mar 2010 18:26:34 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: , Message-ID: Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. What is the point of these functions Fold? Why does it have these various "cost" computations? Granted, they are rough. Should we beef up m3middle? m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. m3back is correct or extremely close to correct as far as I know. It is missing atomic operations for 8, 16, 64 bit operands. (PPC32 seems unable to support 64bit atomics btw.) I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. - Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 13:03:18 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] comparisons vs. subranges Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sat Mar 13 20:23:33 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sat, 13 Mar 2010 14:23:33 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: , Message-ID: On 13 Mar 2010, at 13:26, Jay K wrote: > Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. > > > What is the point of these functions Fold? So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. > Why does it have these various "cost" computations? Which cost computations are you referring to? > Granted, they are rough. > > > Should we beef up m3middle? That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. > m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. > It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. > It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. Designing a decent optimising middle-end takes significant time and effort, and will require more thought. > m3back is correct or extremely close to correct as far as I know. > It is missing atomic operations for 8, 16, 64 bit operands. > (PPC32 seems unable to support 64bit atomics btw.) Correct. > I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. > > > I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Sat, 13 Mar 2010 13:03:18 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] comparisons vs. subranges > > Jay, > > I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 13 Mar 2010, at 05:19, Jay K wrote: > > <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; > <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; > > > Seems to me, the front end should notice these. > The first should always be TRUE. > And possibly, possibly warn. > The second should always be FALSE. > And possibly, possibly warn. > > "Generic" programming often hits this sort of thing though, a good reason not to warn. > Programmer might also be working with existing code and have changed INTEGER to CARDINAL. > Or be defending against future maintainers changing CARDINAL to INTEGER. > > > The backend isn't give enough information, because CARDINAL = INTEGER as far > as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. > The only way to get unsigned types is to use ADDRESS from what I see. > > > - Jay > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sat Mar 13 20:45:29 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sat, 13 Mar 2010 14:45:29 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: , Message-ID: <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Jay, I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 14:23, Tony Hosking wrote: > On 13 Mar 2010, at 13:26, Jay K wrote: > >> Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. >> >> >> What is the point of these functions Fold? > > So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. > >> Why does it have these various "cost" computations? > > Which cost computations are you referring to? > >> Granted, they are rough. >> >> >> Should we beef up m3middle? > > That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. > >> m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. >> It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. >> It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. > > Designing a decent optimising middle-end takes significant time and effort, and will require more thought. > >> m3back is correct or extremely close to correct as far as I know. >> It is missing atomic operations for 8, 16, 64 bit operands. >> (PPC32 seems unable to support 64bit atomics btw.) > > Correct. > >> I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. >> >> >> I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. >> >> >> - Jay >> >> >> From: hosking at cs.purdue.edu >> Date: Sat, 13 Mar 2010 13:03:18 -0500 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] comparisons vs. subranges >> >> Jay, >> >> I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. >> >> Antony Hosking | Associate Professor | Computer Science | Purdue University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> On 13 Mar 2010, at 05:19, Jay K wrote: >> >> <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; >> <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; >> >> >> Seems to me, the front end should notice these. >> The first should always be TRUE. >> And possibly, possibly warn. >> The second should always be FALSE. >> And possibly, possibly warn. >> >> "Generic" programming often hits this sort of thing though, a good reason not to warn. >> Programmer might also be working with existing code and have changed INTEGER to CARDINAL. >> Or be defending against future maintainers changing CARDINAL to INTEGER. >> >> >> The backend isn't give enough information, because CARDINAL = INTEGER as far >> as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. >> The only way to get unsigned types is to use ADDRESS from what I see. >> >> >> - Jay >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmuysers at hotmail.com Sat Mar 13 23:20:07 2010 From: dmuysers at hotmail.com (Dirk Muysers) Date: Sat, 13 Mar 2010 23:20:07 +0100 Subject: [M3devel] bandwith Message-ID: I don't know if other people have the same feeling, but I think that Messrs. Krell and Hosking should keep their polemics to themselves and let the community partake only in the result of a constructive consensus. As a USER of the m3 compiler on windows, I find the recent developments scary, so I prefer to stick to my installed rather old release (5.3.1 if memory serves, BTW the version number should figure at a prominent place in every release.) I am afraid that M3 is going the way of fubar and I am slowly loosing interest. To make it work for all these different platforms seems a herculean task and rather pointless endeavour. Why not use all that energy to try a radically disruptive strategy such as a platform agnostic intermediate form targeting a JIT. Let me apologize in advance to the whole community for the present fit of an old men's anger. d. muysers -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Sun Mar 14 02:54:41 2010 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sun, 14 Mar 2010 01:54:41 +0000 (GMT) Subject: [M3devel] bandwith In-Reply-To: Message-ID: <562452.30843.qm@web23608.mail.ird.yahoo.com> Hi all: I think is an interesting topic, however I disagree with you in the old version mentioned because since then CM3 has evolved a lot in threading, garbage collection and gcc based backend packages, just to name a few. I however seem to recall some discussions, but unfortunately there wasn't sometimes an ending point (i.e my discussion), however without more participation it really becomes boring at one point I expect to be fully honest I do have fun compiling things but probably the community lacks more understanding of the system itself or of the current issues to work in that which needs to be done. Track system by itself is great but probably we haven't used yet in its full potential and that's the reason I waited for so long to report things that would be useful before in ticket trac system (nor I did have the idea to ask you). Current situation in this matter is going better I think. Therefore there could be plenty of opportunities of things to work on in a fashionable way if we may. I don't know but I think the current stability should be the worrisome and the history of this could go back to C interfaces and son on. I haven't asked yet where are the Unix INTERFACE constants and members which are need for packages that before where compiling that use this low-level style like browsers webcard, postcard, Deckscape and Webscape and m3-mail packages and m3-lectern which I could get to compile some time before this Unix changes. However I know that some bugs they had I expect to be gone because this changes brought many other improvements to overall programming environment. Besides this, software engineering tools could be made aware of the project like this one done for pm3: http://libresoft.dat.escet.urjc.es/cvsanal/pm3-cvs/index.php?menu=Index (the manual of it on http://gsyc.es/~carlosgc/files/cvsanaly.pdf software on http://tools.libresoft.es/cvsanaly?rev=1248394389) I know there is also a Modula-3 aware re engineering tools it was done by Peter Klien at Aachen University I know there was supossed to be a working prototype at some point available for download http://pi.informatik.uni-siegen.de/stt/23_3/05_Dissertationen/klein.pdf This kind of tools could be managed to get integrated with M3clipse plugin work done by Bert Laverman which just lacks semantic analysis, and cm3ide which may bundle cvsup and or dcvs as a way of distributed software configuration management and development environment perhaps with Elego Compact (and don't forget Obliq packages, I remember a Professor told me that it competed at some point with JavaScript to be scripting language for the web. It has even type inference algorithm now so who knows why isn't used in our community, I know I will try to get into it but it needs time; there are some examples in Professor's courses that could help to this, besides other things of M3 system level demo applications too) Thanks for your comments I hope this helps a bit --- El s?b, 13/3/10, Dirk Muysers escribi?: > De: Dirk Muysers > Asunto: [M3devel] bandwith > Para: m3devel at elegosoft.com > Fecha: s?bado, 13 de marzo, 2010 17:20 > > > > > > > > I don't know if other > people have the same feeling, > but I think that Messrs. Krell and Hosking > should keep their polemics to > themselves and let the > community partake only > in the result of > a constructive consensus. > > > As a USER of the m3 > compiler on windows, I > find the recent developments scary, so I prefer > to > stick to my installed > rather old release (5.3.1 if > memory serves, BTW the version number should > figure at a prominent > place in every > release.) I am afraid that M3 is going the way of fubar and > I > am slowly loosing > interest. > > To make it work for all > these different platforms > seems a herculean task > and rather > pointless > endeavour. Why not use all > that energy to > try a radically > disruptive strategy such as > a platform > agnostic intermediate form > targeting a > JIT. > > Let > me apologize in advance to the whole > community for the present fit of an old men's > anger. > d. > muysers > From jay.krell at cornell.edu Sun Mar 14 04:52:15 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 03:52:15 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: Hm. You mean, like I'm not supposed to be able to say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (a < b) => IO.Put("true\n"); | (a > b) => IO.Put("false\n"); END; but I can say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); END; (I'd like to be able to say FIRST(a), etc., but I don't think it is allowed.) "constants" kind of seem like an optimization themselves. Other than efficiency concerns, they could all be variables initialized by module initializers, that the frontend doesn't let you write to. Including de-optimizations like runtime allocation/sizing of arrays based on const/non-const sizes. Look at how I changed const to var already in m3core/unix, and then had to change CASE to if/elseif ladders. It's a minor matter of what the language lets you say, among various basically equivalent forms. The compiler could allow case to target non-constants. It'd just mainly be slower. There is the matter of CASE arms can't overlap, where if/elseif ladders can. Because CASE is conceptually compared all at once where if/elseif is sequential. Ok. I see a few options at least. m3middle could look basically like m3front, but with various checks like type checking removed and assumed true. This would be an arduous task of duplication. M3CG.i3 Var and Target.Int could gain the following fields (or methods): min_valid := FALSE; max_valid := FALSE; min := TInt.Zero; max := TInt.Zero; m3front could fill them in a few cases. Initially none, but then a few as they are discovered to be easy, correct, efficient. For constants, min = max value. For subranges, min/max come the type. The backends could use them (if valid) and possibly transform them. e.g. MIN(a,b) with valid min/max yields an expression where min is the min of the two mins, max is the min of the two maxes. MIN([0..1],[2..3]) < 2 => TRUE. Probably more correct would be: PROCEDURE declare_enum (xx: T; t: TypeUID; n_elts: INTEGER; s: BitSize) = PROCEDURE declare_subrange (xx: T; t, domain: TypeUID; READONLY min, max: Target.Int; s: BitSize) => already exist and then PROCEDURE load (xx: T; v: Var; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_indirect (xx: T; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_integer (xx: T; t: IType; READONLY i: Target.Int) = PROCEDURE load_ordered (xx: T; t: MType; u: ZType; order: MemoryOrder) = etc. should all take an optional TypeUID. The second proposal I've thought a bit more about and it seems very easy. The last idea seems cleaner, but with possibly signifiant downside in that e.g. Word.T is not a subrange. There's also some unclarity with respect to Word.LT(expression, -1); -1 is a valid Word interpreted as a large number. One would want to be careful not to break that. I believe this is related to what you were saying, where operations are typed, not values. It could be that every operation needs min/max or typeuid (pairs of them for binary operations). Can we take a stab at something like this? This might also serve to replace some of what m3back does with constants already, since constants would fall out of subranges of length 1. Thanks, -Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 14:45:29 -0500 To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com; jay.krell at cornell.edu Subject: Re: [M3devel] comparisons vs. subranges Jay, I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 14:23, Tony Hosking wrote: On 13 Mar 2010, at 13:26, Jay K wrote: Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. What is the point of these functions Fold? So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. Why does it have these various "cost" computations? Which cost computations are you referring to? Granted, they are rough. Should we beef up m3middle? That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. Designing a decent optimising middle-end takes significant time and effort, and will require more thought. m3back is correct or extremely close to correct as far as I know. It is missing atomic operations for 8, 16, 64 bit operands. (PPC32 seems unable to support 64bit atomics btw.) Correct. I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. - Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 13:03:18 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] comparisons vs. subranges Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 05:31:58 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 04:31:58 +0000 Subject: [M3devel] bandwith In-Reply-To: <562452.30843.qm@web23608.mail.ird.yahoo.com> References: , <562452.30843.qm@web23608.mail.ird.yahoo.com> Message-ID: Arg. Change does not equal instability. Status quo does not equal better. Do you want 64bit LONGINT? Maybe, maybe not. Do you want every integer comparison to needlessly use a temporary on the stack and extra instructions? Maybe, maybe not. Do you want the tables in hand.c statically linked? Maybe, maybe not. They caused quite a headache albeit briefly, when we weren't statically linking them. Standalone apps worked fine, dynamically linked exhibited odd behavior. The odd behavior was "just" gui drawn incorrectly, but was actually indicative of deep problems. Do you want dead buggy code sitting around, waiting to be silently used as m3front changes? (e.g. some forms of extract look wrong). A giant lock in ThreadWin32? vs. the simple pattern documented on the web and what Java uses? Granted, I removed the threadpool, maybe too much simplicity and not enough efficiency? (But have you noticed how much ThreadWin32.m3 did churn through the years anyway?) Can you point to actual bugs? Granted, I don't have specifics in many cases either, such as the bad perf of the giant lock, but everyone knows about them. The system builds itself. I probably rest too much on that, granted. m3-sys/m3tests does pretty well (some stuff does need looking at). I try to add more test coverage when I make changes, but I am lazy, granted. Some non-zero level of progress is presumably desired. The Unix interface was painful to maintain and buggy and tended toward unsafety but claiming safety. It is much improved now. Much safer, much easier to port. A little less efficient. Sort off less amenable to cross building, though really hand.c and lack of an assembler and linker are enough to kill cross builds. The system has never as far I know been really cross buildable, without gcc/binutils support -- has always needed an assembler and linker, and usually needed a C compiler. (NT386 doesn't need an assembler). Just because interface Unix allowed for compiling more code in the past did not make it better or more correct. In fact possibly the opposite. Granted, we should try have it all -- correctness, features, maintability, efficiency. There was a lot of cruft in there getting copied from port to port without being revisited. Stuff having to do with Linux 1.x was in various ports. Every major version of FreeBSD implied extra work. We had FreeBSD, FreeBSD2, FreeBSD3, FreeBSD4. (There are still problems here actually, FreeBSD7 64bit I believe broke the ABI in networking, and we still have the related structs I think -- there is still bad stuff in interface Unix, just a lot less now. I still want to whittle it down more.) There is still code in the system that redeclares struct tm incorrectly, attempting to workaround interface Unix deficiency, when in fact interface Unix was correctly limited (the fields at the end, and perhaps the order of the fields). In the web browsers as I recall (which never seemed to work much in the first place, but they did/do build and come up). > > agnostic intermediate form targeting a JIT. Other than NT386, our portability rests on gcc and Posix, which are very portable with very little effort. I haven't seen any JIT that is as portable. For example there are a few systems we can't run Hudson on because it uses Java. e.g. MIPS64_OPENBSD, PPC32_OPENBSD I would actually like to generate C to gain even more portability. Granted, JIT has the nice property of target-independent distributable binaries. "Package building" matrix could be cut down. Generating C can get you target-independent source distributions, as most things use. (Granted, compilers are somewhat special, except that the world long ago got past not having a C compiler). Targeting a JIT will probably be way more work than anyone has put into this system in over a decade. Still, not a bad idea. It lets you reuse other people's code generators similarly to how we reuse gcc. Java has been made portable to any processor, running Linux. But not yet to other operating systems. I think without too much work, we could use our exiting IL that parse.c reads. It'd just need more context in places, to avoid a need for globals. You could just write an interpreter. It wouldn't be fast, but it would gain more portability probably. Probably not the best idea though. Also, the Java JIT, as one example, is among the more portable, and among the more difficult to target. It doesn't model a CPU as you might expect. It models more so the Java programming language. Which includes non-optional safety. You'd have better luck probably with mono/CLR, not sure as to the level of portability. Mono is pretty widely ported though. Really though, gcc+Posix provides a huge amount of portability with very little effort on our part. The larger work is really obtaining, housing, powering, installing the various systems. If you go the C/portable-JIT route, what you really do is you just stop testing the various targets, for better and worse, probably perfectly ok. Just as nobody tests hello world on every system. They just know it works because it is so portable. > webcard, postcard, Deckscape and Webscape and m3-mail packages and m3-lectern They don't compile? > > rather old release (5.3.1 if memory serves, You can keep using it? Maybe compare that version to current and see if there are particular problems? > > BTW the version number should figure at a prominent place in every release. cm3 -version? As to where Tony and I should argue, well, that's not a generally answerable question. I find that a conversation involving just two people not ideal, no matter who they are. Being under outside scrutiny increases civility, sometimes add more than two opinions, etc. I grant that m3 is such a small community that we don't have an m3-users vs. m3-devel group, and m3-devel is woefully small, and the two groups a bit inappropriately mixed. - Jay > Date: Sun, 14 Mar 2010 01:54:41 +0000 > From: dabenavidesd at yahoo.es > To: m3devel at elegosoft.com; dmuysers at hotmail.com > Subject: Re: [M3devel] bandwith > > Hi all: > I think is an interesting topic, however I disagree with you in the old version mentioned because since then CM3 has evolved a lot in threading, garbage collection and gcc based backend packages, just to name a few. > I however seem to recall some discussions, but unfortunately there wasn't sometimes an ending point (i.e my discussion), however without more participation it really becomes boring at one point I expect to be fully honest I do have fun compiling things but probably the community lacks more understanding of the system itself or of the current issues to work in that which needs to be done. Track system by itself is great but probably we haven't used yet in its full potential and that's the reason I waited for so long to report things that would be useful before in ticket trac system (nor I did have the idea to ask you). Current situation in this matter is going better I think. > Therefore there could be plenty of opportunities of things to work on in a fashionable way if we may. > I don't know but I think the current stability should be the worrisome and the history of this could go back to C interfaces and son on. I haven't asked yet where are the Unix INTERFACE constants and members which are need for packages that before where compiling that use this low-level style like browsers webcard, postcard, Deckscape and Webscape and m3-mail packages and m3-lectern which I could get to compile some time before this Unix changes. > However I know that some bugs they had I expect to be gone because this changes brought many other improvements to overall programming environment. > Besides this, software engineering tools could be made aware of the project like this one done for pm3: > http://libresoft.dat.escet.urjc.es/cvsanal/pm3-cvs/index.php?menu=Index > (the manual of it on http://gsyc.es/~carlosgc/files/cvsanaly.pdf software on http://tools.libresoft.es/cvsanaly?rev=1248394389) > I know there is also a Modula-3 aware re engineering tools it was done by Peter Klien at Aachen University I know there was supossed to be a working prototype at some point available for download > http://pi.informatik.uni-siegen.de/stt/23_3/05_Dissertationen/klein.pdf > This kind of tools could be managed to get integrated with M3clipse plugin work done by Bert Laverman which just lacks semantic analysis, and cm3ide which may bundle cvsup and or dcvs as a way of distributed software configuration management and development environment perhaps with Elego Compact (and don't forget Obliq packages, I remember a Professor told me that it competed at some point with JavaScript to be scripting language for the web. It has even type inference algorithm now so who knows why isn't used in our community, I know I will try to get into it but it needs time; there are some examples in Professor's courses that could help to this, besides other things of M3 system level demo applications too) > Thanks for your comments I hope this helps a bit > > > --- El s?b, 13/3/10, Dirk Muysers escribi?: > > > De: Dirk Muysers > > Asunto: [M3devel] bandwith > > Para: m3devel at elegosoft.com > > Fecha: s?bado, 13 de marzo, 2010 17:20 > > > > > > > > > > > > > > > > I don't know if other > > people have the same feeling, > > but I think that Messrs. Krell and Hosking > > should keep their polemics to > > themselves and let the > > community partake only > > in the result of > > a constructive consensus. > > > > > > As a USER of the m3 > > compiler on windows, I > > find the recent developments scary, so I prefer > > to > > stick to my installed > > rather old release (5.3.1 if > > memory serves, BTW the version number should > > figure at a prominent > > place in every > > release.) I am afraid that M3 is going the way of fubar and > > I > > am slowly loosing > > interest. > > > > To make it work for all > > these different platforms > > seems a herculean task > > and rather > > pointless > > endeavour. Why not use all > > that energy to > > try a radically > > disruptive strategy such as > > a platform > > agnostic intermediate form > > targeting a > > JIT. > > > > Let > > me apologize in advance to the whole > > community for the present fit of an old men's > > anger. > > d. > > muysers > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 06:01:24 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 05:01:24 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: Hm. How about elsewhere m3front? Optimizing middle end not necessarily a different package? (esp. given that m3front is already big enough to afford being organized into directories: m3front/src/middle, m3front/src/target-independent-optimizations?) Maybe elsewhere in the two files I changed? Prep or PrepBR or Compile instead of Fold? I don't understand this Prep vs. PrepBR business. I found out that "BR" stands for branch, but that doesn't explain it me. Also PrepBR and Compile look very similar. I also noticed that Fold gets called twice (I had some RTIO in it) but didn't look into why. Surely doing it in PrepBR or Compile is correct or CG.m3 is correct, reasonable, efficient, maintainable? I can see why Fold would be wrong -- allowing code to compile, with a reasonable meaning, but that isn't supposed to. Or in CG.m3, which is conceptually "the bottom" of m3front, any two adjacent pieces can be considered merged into one layer. I'll look at both of those options later. I think "subrange analysis" if done enough might yield efficiencies in real code. Though m3front might already do enough of it -- e.g. the fact that you can't modify a FOR loop index affords some efficiencies, that are probably already taken into account. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 03:52:15 +0000 Hm. You mean, like I'm not supposed to be able to say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (a < b) => IO.Put("true\n"); | (a > b) => IO.Put("false\n"); END; but I can say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); END; (I'd like to be able to say FIRST(a), etc., but I don't think it is allowed.) "constants" kind of seem like an optimization themselves. Other than efficiency concerns, they could all be variables initialized by module initializers, that the frontend doesn't let you write to. Including de-optimizations like runtime allocation/sizing of arrays based on const/non-const sizes. Look at how I changed const to var already in m3core/unix, and then had to change CASE to if/elseif ladders. It's a minor matter of what the language lets you say, among various basically equivalent forms. The compiler could allow case to target non-constants. It'd just mainly be slower. There is the matter of CASE arms can't overlap, where if/elseif ladders can. Because CASE is conceptually compared all at once where if/elseif is sequential. Ok. I see a few options at least. m3middle could look basically like m3front, but with various checks like type checking removed and assumed true. This would be an arduous task of duplication. M3CG.i3 Var and Target.Int could gain the following fields (or methods): min_valid := FALSE; max_valid := FALSE; min := TInt.Zero; max := TInt.Zero; m3front could fill them in a few cases. Initially none, but then a few as they are discovered to be easy, correct, efficient. For constants, min = max value. For subranges, min/max come the type. The backends could use them (if valid) and possibly transform them. e.g. MIN(a,b) with valid min/max yields an expression where min is the min of the two mins, max is the min of the two maxes. MIN([0..1],[2..3]) < 2 => TRUE. Probably more correct would be: PROCEDURE declare_enum (xx: T; t: TypeUID; n_elts: INTEGER; s: BitSize) = PROCEDURE declare_subrange (xx: T; t, domain: TypeUID; READONLY min, max: Target.Int; s: BitSize) => already exist and then PROCEDURE load (xx: T; v: Var; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_indirect (xx: T; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_integer (xx: T; t: IType; READONLY i: Target.Int) = PROCEDURE load_ordered (xx: T; t: MType; u: ZType; order: MemoryOrder) = etc. should all take an optional TypeUID. The second proposal I've thought a bit more about and it seems very easy. The last idea seems cleaner, but with possibly signifiant downside in that e.g. Word.T is not a subrange. There's also some unclarity with respect to Word.LT(expression, -1); -1 is a valid Word interpreted as a large number. One would want to be careful not to break that. I believe this is related to what you were saying, where operations are typed, not values. It could be that every operation needs min/max or typeuid (pairs of them for binary operations). Can we take a stab at something like this? This might also serve to replace some of what m3back does with constants already, since constants would fall out of subranges of length 1. Thanks, -Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 14:45:29 -0500 To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com; jay.krell at cornell.edu Subject: Re: [M3devel] comparisons vs. subranges Jay, I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 14:23, Tony Hosking wrote: On 13 Mar 2010, at 13:26, Jay K wrote: Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. What is the point of these functions Fold? So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. Why does it have these various "cost" computations? Which cost computations are you referring to? Granted, they are rough. Should we beef up m3middle? That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. Designing a decent optimising middle-end takes significant time and effort, and will require more thought. m3back is correct or extremely close to correct as far as I know. It is missing atomic operations for 8, 16, 64 bit operands. (PPC32 seems unable to support 64bit atomics btw.) Correct. I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. - Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 13:03:18 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] comparisons vs. subranges Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 06:15:22 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 05:15:22 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: > Prep vs. PrepBR vs. Compile Expr.i3 documents it. (*** phase 4 ****) (* Expressions are compiled in two steps: Prep: emit any code that includes branchs or stores Compile: emit the rest of the code *) PROCEDURE Prep (t: T); PROCEDURE Compile (t: T); (* emits code to evaluate the expression onto the top of stack *) PROCEDURE PrepLValue (t: T; traced: BOOLEAN); PROCEDURE CompileLValue (t: T; traced: BOOLEAN); (* emits code to evaluate 't's L-value into s0.A. *) PROCEDURE CompileAddress (t: T; traced: BOOLEAN); (* emits code to evaluate 't's byte address onto the top of stack. Use PrepLValue to prep these expressions. *) PROCEDURE PrepBranch (t: T; true, false: CG.Label; freq: CG.Frequency); PROCEDURE CompileBranch (t: T; true, false: CG.Label; freq: CG.Frequency); (* emits code to evaluate the expression and conditionally branch to 'true' or 'false' depending on its boolean value. 'freq' is the estimated frequency that the specified branch will be taken. *) - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 05:01:24 +0000 Hm. How about elsewhere m3front? Optimizing middle end not necessarily a different package? (esp. given that m3front is already big enough to afford being organized into directories: m3front/src/middle, m3front/src/target-independent-optimizations?) Maybe elsewhere in the two files I changed? Prep or PrepBR or Compile instead of Fold? I don't understand this Prep vs. PrepBR business. I found out that "BR" stands for branch, but that doesn't explain it me. Also PrepBR and Compile look very similar. I also noticed that Fold gets called twice (I had some RTIO in it) but didn't look into why. Surely doing it in PrepBR or Compile is correct or CG.m3 is correct, reasonable, efficient, maintainable? I can see why Fold would be wrong -- allowing code to compile, with a reasonable meaning, but that isn't supposed to. Or in CG.m3, which is conceptually "the bottom" of m3front, any two adjacent pieces can be considered merged into one layer. I'll look at both of those options later. I think "subrange analysis" if done enough might yield efficiencies in real code. Though m3front might already do enough of it -- e.g. the fact that you can't modify a FOR loop index affords some efficiencies, that are probably already taken into account. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 03:52:15 +0000 Hm. You mean, like I'm not supposed to be able to say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (a < b) => IO.Put("true\n"); | (a > b) => IO.Put("false\n"); END; but I can say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); END; (I'd like to be able to say FIRST(a), etc., but I don't think it is allowed.) "constants" kind of seem like an optimization themselves. Other than efficiency concerns, they could all be variables initialized by module initializers, that the frontend doesn't let you write to. Including de-optimizations like runtime allocation/sizing of arrays based on const/non-const sizes. Look at how I changed const to var already in m3core/unix, and then had to change CASE to if/elseif ladders. It's a minor matter of what the language lets you say, among various basically equivalent forms. The compiler could allow case to target non-constants. It'd just mainly be slower. There is the matter of CASE arms can't overlap, where if/elseif ladders can. Because CASE is conceptually compared all at once where if/elseif is sequential. Ok. I see a few options at least. m3middle could look basically like m3front, but with various checks like type checking removed and assumed true. This would be an arduous task of duplication. M3CG.i3 Var and Target.Int could gain the following fields (or methods): min_valid := FALSE; max_valid := FALSE; min := TInt.Zero; max := TInt.Zero; m3front could fill them in a few cases. Initially none, but then a few as they are discovered to be easy, correct, efficient. For constants, min = max value. For subranges, min/max come the type. The backends could use them (if valid) and possibly transform them. e.g. MIN(a,b) with valid min/max yields an expression where min is the min of the two mins, max is the min of the two maxes. MIN([0..1],[2..3]) < 2 => TRUE. Probably more correct would be: PROCEDURE declare_enum (xx: T; t: TypeUID; n_elts: INTEGER; s: BitSize) = PROCEDURE declare_subrange (xx: T; t, domain: TypeUID; READONLY min, max: Target.Int; s: BitSize) => already exist and then PROCEDURE load (xx: T; v: Var; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_indirect (xx: T; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_integer (xx: T; t: IType; READONLY i: Target.Int) = PROCEDURE load_ordered (xx: T; t: MType; u: ZType; order: MemoryOrder) = etc. should all take an optional TypeUID. The second proposal I've thought a bit more about and it seems very easy. The last idea seems cleaner, but with possibly signifiant downside in that e.g. Word.T is not a subrange. There's also some unclarity with respect to Word.LT(expression, -1); -1 is a valid Word interpreted as a large number. One would want to be careful not to break that. I believe this is related to what you were saying, where operations are typed, not values. It could be that every operation needs min/max or typeuid (pairs of them for binary operations). Can we take a stab at something like this? This might also serve to replace some of what m3back does with constants already, since constants would fall out of subranges of length 1. Thanks, -Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 14:45:29 -0500 To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com; jay.krell at cornell.edu Subject: Re: [M3devel] comparisons vs. subranges Jay, I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 14:23, Tony Hosking wrote: On 13 Mar 2010, at 13:26, Jay K wrote: Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. What is the point of these functions Fold? So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. Why does it have these various "cost" computations? Which cost computations are you referring to? Granted, they are rough. Should we beef up m3middle? That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. Designing a decent optimising middle-end takes significant time and effort, and will require more thought. m3back is correct or extremely close to correct as far as I know. It is missing atomic operations for 8, 16, 64 bit operands. (PPC32 seems unable to support 64bit atomics btw.) Correct. I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. - Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 13:03:18 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] comparisons vs. subranges Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 11:43:57 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 10:43:57 +0000 Subject: [M3devel] comparing different types/signedness Message-ID: Tony, is it deliberate in m3front/src/misc/cg.m3 procedure compare that we are often asked to compare different types, even different signedness? It seems a little unusual. I think it really might help to extend TInt to 9 or more bytes so that any pair of values we "really" encounter can be extended to a common type/precision. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 13:36:50 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 12:36:50 +0000 Subject: [M3devel] comparing different types/signedness In-Reply-To: References: Message-ID: I got through my problem here, not a big deal. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu; m3devel at elegosoft.com Date: Sun, 14 Mar 2010 10:43:57 +0000 Subject: [M3devel] comparing different types/signedness Tony, is it deliberate in m3front/src/misc/cg.m3 procedure compare that we are often asked to compare different types, even different signedness? It seems a little unusual. I think it really might help to extend TInt to 9 or more bytes so that any pair of values we "really" encounter can be extended to a common type/precision. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 13:43:02 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 12:43:02 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: Tony how about the attached? It achieves the "same" thing as before. CG is about the lowest level in m3front, therefore akin to any middle end below m3front. The vast bulk of the change is in CG.m3, with a small change in Variable.m3 to pass it the bounds. It took me a while to cope with the mixture of signed and unsigned that the front end throws at its CG.m3. More can be done here. e.g. mod a positive non-zero constant returns 0..n-1. min and max(a,b) has bounds computable from the bounds of a and b. abs returns a positive number, except for the overflow case neg(abs) returns 0 or negative (again with some chance of overflow) I think the change is ok. There is the small matter of how to call GetBounds. I find it a little wierd that some versions of GetBounds return a boolean, some do not. The signed/unsigned cases could be combined down somehow, replacing min/max in some places with zero. Load_addr_of_temp should probably use WITH. Other forms of Load e.g. Load_indirect should probably be changed analogously. There is also a matter of "bounds" for non-ordinal types. For example a set might be a constant. It might be possible to reason about floating point. But I didn't do any this stuf. Just ordinal types. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 05:15:22 +0000 > Prep vs. PrepBR vs. Compile Expr.i3 documents it. (*** phase 4 ****) (* Expressions are compiled in two steps: Prep: emit any code that includes branchs or stores Compile: emit the rest of the code *) PROCEDURE Prep (t: T); PROCEDURE Compile (t: T); (* emits code to evaluate the expression onto the top of stack *) PROCEDURE PrepLValue (t: T; traced: BOOLEAN); PROCEDURE CompileLValue (t: T; traced: BOOLEAN); (* emits code to evaluate 't's L-value into s0.A. *) PROCEDURE CompileAddress (t: T; traced: BOOLEAN); (* emits code to evaluate 't's byte address onto the top of stack. Use PrepLValue to prep these expressions. *) PROCEDURE PrepBranch (t: T; true, false: CG.Label; freq: CG.Frequency); PROCEDURE CompileBranch (t: T; true, false: CG.Label; freq: CG.Frequency); (* emits code to evaluate the expression and conditionally branch to 'true' or 'false' depending on its boolean value. 'freq' is the estimated frequency that the specified branch will be taken. *) - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 05:01:24 +0000 Hm. How about elsewhere m3front? Optimizing middle end not necessarily a different package? (esp. given that m3front is already big enough to afford being organized into directories: m3front/src/middle, m3front/src/target-independent-optimizations?) Maybe elsewhere in the two files I changed? Prep or PrepBR or Compile instead of Fold? I don't understand this Prep vs. PrepBR business. I found out that "BR" stands for branch, but that doesn't explain it me. Also PrepBR and Compile look very similar. I also noticed that Fold gets called twice (I had some RTIO in it) but didn't look into why. Surely doing it in PrepBR or Compile is correct or CG.m3 is correct, reasonable, efficient, maintainable? I can see why Fold would be wrong -- allowing code to compile, with a reasonable meaning, but that isn't supposed to. Or in CG.m3, which is conceptually "the bottom" of m3front, any two adjacent pieces can be considered merged into one layer. I'll look at both of those options later. I think "subrange analysis" if done enough might yield efficiencies in real code. Though m3front might already do enough of it -- e.g. the fact that you can't modify a FOR loop index affords some efficiencies, that are probably already taken into account. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 03:52:15 +0000 Hm. You mean, like I'm not supposed to be able to say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (a < b) => IO.Put("true\n"); | (a > b) => IO.Put("false\n"); END; but I can say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); END; (I'd like to be able to say FIRST(a), etc., but I don't think it is allowed.) "constants" kind of seem like an optimization themselves. Other than efficiency concerns, they could all be variables initialized by module initializers, that the frontend doesn't let you write to. Including de-optimizations like runtime allocation/sizing of arrays based on const/non-const sizes. Look at how I changed const to var already in m3core/unix, and then had to change CASE to if/elseif ladders. It's a minor matter of what the language lets you say, among various basically equivalent forms. The compiler could allow case to target non-constants. It'd just mainly be slower. There is the matter of CASE arms can't overlap, where if/elseif ladders can. Because CASE is conceptually compared all at once where if/elseif is sequential. Ok. I see a few options at least. m3middle could look basically like m3front, but with various checks like type checking removed and assumed true. This would be an arduous task of duplication. M3CG.i3 Var and Target.Int could gain the following fields (or methods): min_valid := FALSE; max_valid := FALSE; min := TInt.Zero; max := TInt.Zero; m3front could fill them in a few cases. Initially none, but then a few as they are discovered to be easy, correct, efficient. For constants, min = max value. For subranges, min/max come the type. The backends could use them (if valid) and possibly transform them. e.g. MIN(a,b) with valid min/max yields an expression where min is the min of the two mins, max is the min of the two maxes. MIN([0..1],[2..3]) < 2 => TRUE. Probably more correct would be: PROCEDURE declare_enum (xx: T; t: TypeUID; n_elts: INTEGER; s: BitSize) = PROCEDURE declare_subrange (xx: T; t, domain: TypeUID; READONLY min, max: Target.Int; s: BitSize) => already exist and then PROCEDURE load (xx: T; v: Var; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_indirect (xx: T; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_integer (xx: T; t: IType; READONLY i: Target.Int) = PROCEDURE load_ordered (xx: T; t: MType; u: ZType; order: MemoryOrder) = etc. should all take an optional TypeUID. The second proposal I've thought a bit more about and it seems very easy. The last idea seems cleaner, but with possibly signifiant downside in that e.g. Word.T is not a subrange. There's also some unclarity with respect to Word.LT(expression, -1); -1 is a valid Word interpreted as a large number. One would want to be careful not to break that. I believe this is related to what you were saying, where operations are typed, not values. It could be that every operation needs min/max or typeuid (pairs of them for binary operations). Can we take a stab at something like this? This might also serve to replace some of what m3back does with constants already, since constants would fall out of subranges of length 1. Thanks, -Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 14:45:29 -0500 To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com; jay.krell at cornell.edu Subject: Re: [M3devel] comparisons vs. subranges Jay, I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 14:23, Tony Hosking wrote: On 13 Mar 2010, at 13:26, Jay K wrote: Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. What is the point of these functions Fold? So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. Why does it have these various "cost" computations? Which cost computations are you referring to? Granted, they are rough. Should we beef up m3middle? That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. Designing a decent optimising middle-end takes significant time and effort, and will require more thought. m3back is correct or extremely close to correct as far as I know. It is missing atomic operations for 8, 16, 64 bit operands. (PPC32 seems unable to support 64bit atomics btw.) Correct. I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. - Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 13:03:18 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] comparisons vs. subranges Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 1.txt URL: From rodney_bates at lcwb.coop Sun Mar 14 16:20:29 2010 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 14 Mar 2010 10:20:29 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <20100313182924.GA14329@topoi.pooq.com> References: <20100313182924.GA14329@topoi.pooq.com> Message-ID: <4B9CFEBD.8050309@lcwb.coop> hendrik at topoi.pooq.com wrote: > > Wasn't there a discussion a while ago about subranges out-of-bounds not > being a safety problem? (Or was it arithmetic overflow?) This > optimisation might well cause a quite hard-to-find bug if we don't > guarantee subrange integrity. Subrange is a safety problem. Overflow is not, although it can be a valuable way for the language/implementation to help find bugs. Most of the definitions of safety are neither very useful nor consistent with common, informal usage. My definition is that safety means everything that can happen can be explained and understood using only the concepts of the programming language. You don't have to resort to knowing about machine level stuff, especially bit representations and the fact that things that are entirely autonomous in the language (e.g. separately declared variables) actually occupy the same homogeneous memory, along with machine code and all sorts of other stuff. So, if LAST(INTEGER)+LAST(INTEGER) overflows, even if it is undefined by the language what happens, all the possibilities are still comprehensible in terms of the language. Either you get some value that is a member of INTEGER, or an exception, all of which are language concepts. But if you could assign 16_FF to a variable of type [0..20], you get something that can only be understood by looking at the machine-level representation. Most likely, it is a bit pattern that would represent a value that is not a member of the variable's type. This is a safely problem. BTW, it's also implementation- and target-dependent. > > -- hendrik > From rodney_bates at lcwb.coop Sun Mar 14 16:58:21 2010 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 14 Mar 2010 10:58:21 -0500 Subject: [M3devel] bandwith In-Reply-To: References: Message-ID: <4B9D079D.1060207@lcwb.coop> Dirk Muysers wrote: > > To make it work for all these different platforms seems a herculean task > and rather pointless > endeavour. Why not use all that energy to try a radically disruptive > strategy such as a platform > agnostic intermediate form targeting a JIT. > I have been a little way down this road at least three times, with at least two different languages, and it turns out not to be what it would appear at first glance, at least for the JVM. Java is a mostly object-oriented language, meaning the non-object-oriented types and operations are very limited. Most of the interesting stuff can be done only with full-blown heap allocated objects and dispatching methods. Even the equivalents of records and arrays are this way inb Java. When you don't need this full generality and power, you pay big performance penalties for heap allocation, and this is worse and much deeper than the performance penalties of an interpretive implementation. (which, of course, a JIT code generator partially avoids, and a traditional source-to-machine compiler avoids altogether, all with no language changes.) Many languages, Modula-3 included, give this generality for when you need it, but also give you lighter weight alternatives, for use when appropriate. So when you try to implement some other language using JVM, you soon discover that JVM lacks a lot of operations needed for the not-so- impoverished non-object-oriented types of the language. Whether or how badly the .net/mono virtual machine suffers from this, I don't know. It apparently was designed to support a much wider range of languages, but I seem to recall hearing some discussions that suggested it had some of the same limitations. Of course, we could invent our own virtual machine (M-code? MVM?) ;-). This whole approach is of course, not very suitable for systems programming, one of Modula-3's strengths. > d. muysers From rodney_bates at lcwb.coop Sun Mar 14 17:54:58 2010 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 14 Mar 2010 11:54:58 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: <4B9D14E2.6020403@lcwb.coop> Jay K wrote: > Hm. You mean, like I'm not supposed to be able to say: > > > PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= > BEGIN > CASE bool OF > | (a < b) => IO.Put("true\n"); > | (a > b) => IO.Put("false\n"); > END; > No, because a and b are not constants, and thus neither is a < b. Use an IF ladder for this. > > but I can say: > > > PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= > BEGIN > CASE bool OF > | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); > | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); > END; Yes. You can apply FIRST to a(non-open) array or ordinal _type_. > > (I'd like to be able to say FIRST(a), etc., but I don't think it is > allowed.) > Yes, or rather no. Yes, it's not allowed. Yes, we have no bananas. You can apply FIRST to a _variable_ of array type and it will be legal and (if not an open array), constant. You can't apply FIRST to a variable of ordinal type. While probably not very useful, it has always seemed to me to be a bit inconsistent that you can't. It might be useful in some kind of generated, generic, or otherwise parameterized code. If you intend these examples just to talk about language rules, OK. If examples of how to write actual code, they seem convoluted to me. They are better candidates for IF ladders. When the type of the expression is BOOLEAN, probably a single IF statement, with ELSE, is in order. The only situation I can think of where I would code something like this is if I wanted to force a compile-time error if the two expressions in the two alternatives ever became equal during maintenance, because someone changed one of the constants or types in a way that violated some algorithmic constraint. Or maybe just to emphasize that the case label expression should be constant and get the compiler to verify that. From hosking at cs.purdue.edu Sun Mar 14 19:28:56 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 14 Mar 2010 14:28:56 -0400 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: Correct. On 13 Mar 2010, at 22:52, Jay K wrote: > Hm. You mean, like I'm not supposed to be able to say: > > > PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= > BEGIN > CASE bool OF > | (a < b) => IO.Put("true\n"); > | (a > b) => IO.Put("false\n"); > END; > > > but I can say: > > > PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= > BEGIN > CASE bool OF > | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); > | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); > END; > > (I'd like to be able to say FIRST(a), etc., but I don't think it is allowed.) > > > > "constants" kind of seem like an optimization themselves. > Other than efficiency concerns, they could all be variables > initialized by module initializers, that the frontend doesn't > let you write to. Including de-optimizations like runtime > allocation/sizing of arrays based on const/non-const sizes. > > > Look at how I changed const to var already in m3core/unix, and then > had to change CASE to if/elseif ladders. > It's a minor matter of what the language lets you say, > among various basically equivalent forms. > The compiler could allow case to target non-constants. > It'd just mainly be slower. > There is the matter of CASE arms can't overlap, where if/elseif ladders can. > Because CASE is conceptually compared all at once where if/elseif > is sequential. > > > Ok. > > > > I see a few options at least. > > > m3middle could look basically like m3front, but with various checks like type checking removed > and assumed true. This would be an arduous task of duplication. > > M3CG.i3 Var and Target.Int could gain the following fields (or methods): > > > min_valid := FALSE; > max_valid := FALSE; > min := TInt.Zero; > max := TInt.Zero; > > > m3front could fill them in a few cases. > Initially none, but then a few as they are discovered to be easy, correct, efficient. > For constants, min = max value. > For subranges, min/max come the type. > The backends could use them (if valid) and possibly transform them. > e.g. MIN(a,b) with valid min/max yields an expression > where min is the min of the two mins, max is the min of the two maxes. > > > MIN([0..1],[2..3]) < 2 => TRUE. > > > Probably more correct would be: > > PROCEDURE declare_enum (xx: T; t: TypeUID; n_elts: INTEGER; s: BitSize) = > > PROCEDURE declare_subrange (xx: T; t, domain: TypeUID; > READONLY min, max: Target.Int; > s: BitSize) > > => already exist > > > and then > > > PROCEDURE load (xx: T; v: Var; o: ByteOffset; t: MType; u: ZType) = > PROCEDURE load_indirect (xx: T; o: ByteOffset; t: MType; u: ZType) = > PROCEDURE load_integer (xx: T; t: IType; READONLY i: Target.Int) = > PROCEDURE load_ordered (xx: T; t: MType; u: ZType; order: MemoryOrder) = > etc. > > > should all take an optional TypeUID. > > > The second proposal I've thought a bit more about and it seems very easy. > The last idea seems cleaner, but with possibly signifiant downside > in that e.g. Word.T is not a subrange. > > There's also some unclarity with respect to > Word.LT(expression, -1); > > -1 is a valid Word interpreted as a large number. > One would want to be careful not to break that. > I believe this is related to what you were saying, where operations > are typed, not values. > > > It could be that every operation needs min/max or typeuid (pairs > of them for binary operations). > > > Can we take a stab at something like this? > > > This might also serve to replace some of what m3back does with constants already, > since constants would fall out of subranges of length 1. > > > Thanks, > -Jay > > > From: hosking at cs.purdue.edu > Date: Sat, 13 Mar 2010 14:45:29 -0500 > To: hosking at cs.purdue.edu > CC: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: Re: [M3devel] comparisons vs. subranges > > Jay, > > I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 13 Mar 2010, at 14:23, Tony Hosking wrote: > > On 13 Mar 2010, at 13:26, Jay K wrote: > > Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. > > > What is the point of these functions Fold? > > So that constants can be manipulated as first-class values in the language. Seehttp://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. > > Why does it have these various "cost" computations? > > Which cost computations are you referring to? > > Granted, they are rough. > > > Should we beef up m3middle? > > That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. > > m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. > It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. > It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. > > Designing a decent optimising middle-end takes significant time and effort, and will require more thought. > > m3back is correct or extremely close to correct as far as I know. > It is missing atomic operations for 8, 16, 64 bit operands. > (PPC32 seems unable to support 64bit atomics btw.) > > Correct. > > I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. > > > I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Sat, 13 Mar 2010 13:03:18 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] comparisons vs. subranges > > Jay, > > I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 13 Mar 2010, at 05:19, Jay K wrote: > > <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; > <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; > > > Seems to me, the front end should notice these. > The first should always be TRUE. > And possibly, possibly warn. > The second should always be FALSE. > And possibly, possibly warn. > > "Generic" programming often hits this sort of thing though, a good reason not to warn. > Programmer might also be working with existing code and have changed INTEGER to CARDINAL. > Or be defending against future maintainers changing CARDINAL to INTEGER. > > > The backend isn't give enough information, because CARDINAL = INTEGER as far > as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. > The only way to get unsigned types is to use ADDRESS from what I see. > > > - Jay > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 19:30:59 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 18:30:59 +0000 Subject: [M3devel] atomics vs. locks.. Message-ID: http://msdn.microsoft.com/en-us/library/ee418650(VS.85).aspx MemoryBarrier was measured as taking 20-90 cycles. InterlockedIncrement was measured as taking 36-90 cycles. Acquiring or releasing a critical section was measured as taking 40-100 cycles. Acquiring or releasing a mutex was measured as taking about 750-2500 cycles perhaps just use critical sections... (I'm very ambivalent about atomics. It is just so darn difficult to do lockless programming. Granted, if you are a systems language, then you need to be able to imlement the locks themselves. Perhaps.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Sun Mar 14 22:20:26 2010 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sun, 14 Mar 2010 21:20:26 +0000 (GMT) Subject: [M3devel] bandwith Message-ID: <646699.86796.qm@web23603.mail.ird.yahoo.com> Hi Rodney: Well, I think the main issue as you point out is the architecture of the VM, as you know JVM is a Java based architecture so many of its structures are suited for that language and probably this is related to a compiler efficiency thing and thus you can say that a big language like Java needs an appropiate VM like JVM. I think that in M3 case what could be more suitable for real use is a SPIN powered vm like itself running in bare hardware with current hardware assisted virtualization such of current leading processors that would be a plus, it would lack supoort for more real machine languages (i.e C), but as it is running in kernel mode could give a better performance. In other architectures it could be run with kernel level virtualization and thus gaining better performance or similar even if its not a complete form of virtualization in bare hardware (by writing for SPIN M3 or other VM based language too and run in kernel mode like a M3 process, like the JVM M3 hosted if so, etc). I think having a kernel level virtual machine could be a gain in performance for some applications written for other machines or languages and or virtual machines or even the same kind of machines in M3 but in other systems (like running a DEC Unix or BSD flawor even with C libraries from inside SPIN). In fact SPIN was ported to NT as a driver for an intelligent NIC, thus allowed to run in bare hardware and at kernel level Thanks in advance, --- El dom, 14/3/10, Rodney M. Bates escribi?: > De: Rodney M. Bates > Asunto: Re: [M3devel] bandwith > Para: m3devel at elegosoft.com > Fecha: domingo, 14 de marzo, 2010 10:58 > > > Dirk Muysers wrote: > >? To make it work for all these different > platforms seems a herculean task and rather pointless > > endeavour. Why not use all that energy to try a > radically disruptive strategy such as a platform > > agnostic intermediate form targeting a JIT. > >? > > I have been a little way down this road at least three > times, with at > least two different languages, and it turns out not to be > what it would > appear at first glance, at least for the JVM. > > Java is a mostly object-oriented language, meaning the > non-object-oriented > types and operations are very limited.? Most of the > interesting stuff can > be done only with full-blown heap allocated objects and > dispatching methods. > Even the equivalents of records and arrays are this way inb > Java.? When you > don't need this full generality and power, you pay big > performance penalties > for heap allocation, and this is worse and much deeper than > the > performance penalties of an interpretive implementation. > (which, of > course, a JIT code generator partially avoids, and a > traditional > source-to-machine compiler avoids altogether, all with no > language > changes.) > > Many languages, Modula-3 included, give this generality for > when you need > it, but also give you lighter weight alternatives, for use > when appropriate. > > So when you try to implement some other language using JVM, > you soon > discover that JVM lacks a lot of operations needed for the > not-so- > impoverished non-object-oriented types of the language. > > Whether or how badly the .net/mono virtual machine suffers > from this, I > don't know.? It apparently was designed to support a > much wider range > of languages, but I seem to recall hearing some discussions > that suggested > it had some of the same limitations. > > Of course, we could invent our own virtual machine (M-code? > MVM?) ;-). > > This whole approach is of course, not very suitable for > systems programming, > one of Modula-3's strengths. > > > > > d. muysers > From jay.krell at cornell.edu Mon Mar 15 03:51:54 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 15 Mar 2010 02:51:54 +0000 Subject: [M3devel] Mx86.m3 vs. stackx86.m3? Message-ID: Does anyone understand and can explain the rationale to put some things in Mx86.m3 and some in Stackx86.m3? It seems arbitrary in many places to me. For exampe, in the release branch, compare M3x86 shift, shift_left, shift_right. shift_left and shift_right do the work themselves, but shift calls stack.doshift. In this I case merged all the code into stack.doshift, with an enum to indicate which of the three types of shifts to do. rotate is still the old way. (I don't yet inline 64bit rotates without constants.) Also, I assume "vstack" means "virtual stack"? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 16 01:54:41 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 00:54:41 +0000 Subject: [M3devel] arbitrary use of LONGINT for testing? target/ABI alteration/generation via command line? Message-ID: Tony et. al. I'm wondering/fishing if you any ideas, like a cm3 command line switch, that might be used to make INTEGER 64bits to increase coverage/testing of 64bit integer support. The problem is *at least* interfacing with external code/files, if not breaking everything. Maybe also a problem around sizeof(INTEGER) vs. sizeof(ADDRESS). Probably such a setting would grow ADDRESS too. Like, maybe stuff like m3core/unix, m3core/win32 should never use plain INTEGER or LONGINT, but "BITS FOR", to allow this to work? Or maybe <* EXTERNAL *> would be the hint? No, <* EXTERNAL *> isn't adequate, due to the case of callbacks. I'd like to somehow, without too much effort, significantly increase testing of LONGINT. Maybe a wierdo platform NT386_64 that does this just for test purposes? I might try that, if "BITS FOR" is enough to actually let it work (interface with C). (or generally, cm3 -test64 would append "_TEST64" to target name, so we could have LINUXLIBC6_TEST64, I386_DARWIN_TEST64, etc.) I can do some initial experiments along this -test64 line, see if it completely blows up or not. A few minutes thought suggests it should work easily and provide value. (There's probably something similar to try around a 16 bit CHAR. vs. BITS 8 FOR CHAR, but I'm not going there..) Such command line driven target/ABI alterations seem like a somewhat good idea. e.g. cm3 -userthreads would append _USERTHREADS. cm3 -extended80 would use an 80 or 96 bit extended on x86 (96 bits for alignment). -extended128 on ppc etc. cm3 -msvc80 could probe around in enviroment for a Visual C++ 8.0 compiler/linker/header/libraries and append "_MSVC80" to target, else fail. I'll just try -test64 for now. The rest aren't as immediately useful. -userthreads would be my next "favorate". Hm. Instead of -test64, how about -64. If integer is 32 bits, change it to 64 and append _64 (and address). If integer is already 64bits, do nothing. This is quite different than gcc's -m64 though. Maybe not good that way. You can see parallels in several older C compilers. Metrowerks for Mac/68K let you chose integer size and either double or extended size (I forget which). That gave you a cross product set of ABIs, each with their own libraries. Similarly you could chose to issue FPU instructions directly, which might have altered the ABI, or maybe the alternate libraries were just faster. Similarly Microsoft and memory models. Various switches changed the size of a pointer and implied a need for different libraries. Current gcc and ppc floating point sizes I believe is a contemporary example. Even sometimes the affect that -pthread has -- chosing different libraries (unnecessary mess imho, but one that seems to persist on HP-UX.) It's kind of an ugly area, to offer too many choices, produces confusion, but the limited important goal of increasing 64bit integer testing seems worth going here. And increasing userthread testing. People have expressed an interest in higher precision floating point, esp. 80 bit on x86, so this maybe a good way to proceed there also. You could also imagine flags -SSE, -SSE2, etc. It is important to only support a very limited number of these flags. Large cross products are not fun to worry about and test. But for example, with SSE, that nets you several targets all at once, without per-target work. (I lose the term "SSE" loosely. Could be I mean "SSE2" or 3 -- whatever is sufficient to largely replace the stack based x87.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 02:00:27 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 15 Mar 2010 20:00:27 -0500 Subject: [M3devel] arbitrary use of LONGINT for testing? target/ABI alteration/generation via command line? In-Reply-To: References: Message-ID: <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu> Let's not right now. We need to get the release out. I suggest a freeze on compiler changes for now. Bug fixes only! Sent from my iPhone On Mar 15, 2010, at 7:54 PM, Jay K wrote: > Tony et. al. I'm wondering/fishing if you any ideas, like a cm3 > command line switch, that might be used to make INTEGER 64bits to > increase coverage/testing of 64bit integer support. > > > The problem is *at least* interfacing with external code/files, if > not breaking everything. > Maybe also a problem around sizeof(INTEGER) vs. sizeof(ADDRESS). > Probably such a setting would grow ADDRESS too. > > > Like, maybe stuff like m3core/unix, m3core/win32 should never use > plain INTEGER or LONGINT, but "BITS FOR", to allow this to work? Or > maybe <* EXTERNAL *> would be the hint? > No, <* EXTERNAL *> isn't adequate, due to the case of callbacks. > > > I'd like to somehow, without too much effort, significantly increase > testing of LONGINT. > > > Maybe a wierdo platform NT386_64 that does this just for test > purposes? > I might try that, if "BITS FOR" is enough to actually let it work > (interface with C). > (or generally, cm3 -test64 would append "_TEST64" to target name, > so we could have LINUXLIBC6_TEST64, I386_DARWIN_TEST64, etc.) > > > I can do some initial experiments along this -test64 line, see if it > completely blows up or not. > A few minutes thought suggests it should work easily and provide > value. > > > (There's probably something similar to try around a 16 bit CHAR. vs. > BITS 8 FOR CHAR, but I'm not going there..) > > > Such command line driven target/ABI alterations seem like a somewhat > good idea. > e.g. cm3 -userthreads would append _USERTHREADS. > cm3 -extended80 would use an 80 or 96 bit extended on x86 (96 bits > for alignment). > -extended128 on ppc > etc. > > > cm3 -msvc80 could probe around in enviroment for a Visual C++ 8.0 > compiler/linker/header/libraries and append "_MSVC80" to target, > else fail. > > > I'll just try -test64 for now. > The rest aren't as immediately useful. > -userthreads would be my next "favorate". > > Hm. Instead of -test64, how about -64. > If integer is 32 bits, change it to 64 and append _64 (and address). > If integer is already 64bits, do nothing. > > > This is quite different than gcc's -m64 though. > Maybe not good that way. > > > You can see parallels in several older C compilers. > > Metrowerks for Mac/68K let you chose integer size and either double > or extended size (I forget which). > That gave you a cross product set of ABIs, each with their own > libraries. > > > Similarly you could chose to issue FPU instructions directly, which > might have altered the ABI, or maybe the alternate libraries were > just faster. > > > Similarly Microsoft and memory models. Various switches changed the > size of a pointer and implied a need for different libraries. > > > Current gcc and ppc floating point sizes I believe is a contemporary > example. > Even sometimes the affect that -pthread has -- chosing different > libraries (unnecessary > mess imho, but one that seems to persist on HP-UX.) > > > It's kind of an ugly area, to offer too many choices, produces > confusion, but the limited important goal of increasing 64bit > integer testing seems worth going here. And increasing userthread > testing. People have expressed an interest in higher precision > floating point, esp. 80 bit on x86, so this maybe a good way to > proceed there also. > You could also imagine flags -SSE, -SSE2, etc. > > > It is important to only support a very limited number of these flags. > Large cross products are not fun to worry about and test. > But for example, with SSE, that nets you several targets all at > once, without per-target work. > > (I lose the term "SSE" loosely. Could be I mean "SSE2" or 3 -- > whatever is sufficient to largely replace the stack based x87.) > > > - Jay From jay.krell at cornell.edu Tue Mar 16 02:41:37 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 01:41:37 +0000 Subject: [M3devel] release status [was something else] In-Reply-To: <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu> References: , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu> Message-ID: Release status as far as I know: - cvsupd hang Reported over a year ago. Needs investigation. Can anyone confirm it with a recent version? And, as Tony asked, with user threads? - NT386 has no 64bit longint in release branch. Partly the point of this question. - Should maybe improve build/packaging to get NT386-VC80.msi, NT386-VC90.msi etc. (not the same as "target/abi alteration via command line" expressed below, though that is a possibility, but separate clean builds for now and/or separate CVS checkouts); really more of a Hudson task now, to have multiple build environments installed and used, I think I already updated the .msi building to append the tag. Olaf? - I believe there is "new" divergence in m3front between head and release -- whatever came along with the TInt changes. Otherwise I'd have to look through trac. There's maybe some stuff not merged from release to head, but that's ok for release. (I need to check which branch Randy fixed the examples in.) I need to re-test the .msis. - Jay > From: hosking at cs.purdue.edu > To: jay.krell at cornell.edu > Subject: Re: [M3devel] arbitrary use of LONGINT for testing? target/ABI alteration/generation via command line? > Date: Mon, 15 Mar 2010 20:00:27 -0500 > CC: m3devel at elegosoft.com > > Let's not right now. We need to get the release out. I suggest a > freeze on compiler changes for now. Bug fixes only! > > Sent from my iPhone > > On Mar 15, 2010, at 7:54 PM, Jay K wrote: > > > Tony et. al. I'm wondering/fishing if you any ideas, like a cm3 > > command line switch, that might be used to make INTEGER 64bits to > > increase coverage/testing of 64bit integer support. > > > > > > The problem is *at least* interfacing with external code/files, if > > not breaking everything. > > Maybe also a problem around sizeof(INTEGER) vs. sizeof(ADDRESS). > > Probably such a setting would grow ADDRESS too. > > > > > > Like, maybe stuff like m3core/unix, m3core/win32 should never use > > plain INTEGER or LONGINT, but "BITS FOR", to allow this to work? Or > > maybe <* EXTERNAL *> would be the hint? > > No, <* EXTERNAL *> isn't adequate, due to the case of callbacks. > > > > > > I'd like to somehow, without too much effort, significantly increase > > testing of LONGINT. > > > > > > Maybe a wierdo platform NT386_64 that does this just for test > > purposes? > > I might try that, if "BITS FOR" is enough to actually let it work > > (interface with C). > > (or generally, cm3 -test64 would append "_TEST64" to target name, > > so we could have LINUXLIBC6_TEST64, I386_DARWIN_TEST64, etc.) > > > > > > I can do some initial experiments along this -test64 line, see if it > > completely blows up or not. > > A few minutes thought suggests it should work easily and provide > > value. > > > > > > (There's probably something similar to try around a 16 bit CHAR. vs. > > BITS 8 FOR CHAR, but I'm not going there..) > > > > > > Such command line driven target/ABI alterations seem like a somewhat > > good idea. > > e.g. cm3 -userthreads would append _USERTHREADS. > > cm3 -extended80 would use an 80 or 96 bit extended on x86 (96 bits > > for alignment). > > -extended128 on ppc > > etc. > > > > > > cm3 -msvc80 could probe around in enviroment for a Visual C++ 8.0 > > compiler/linker/header/libraries and append "_MSVC80" to target, > > else fail. > > > > > > I'll just try -test64 for now. > > The rest aren't as immediately useful. > > -userthreads would be my next "favorate". > > > > Hm. Instead of -test64, how about -64. > > If integer is 32 bits, change it to 64 and append _64 (and address). > > If integer is already 64bits, do nothing. > > > > > > This is quite different than gcc's -m64 though. > > Maybe not good that way. > > > > > > You can see parallels in several older C compilers. > > > > Metrowerks for Mac/68K let you chose integer size and either double > > or extended size (I forget which). > > That gave you a cross product set of ABIs, each with their own > > libraries. > > > > > > Similarly you could chose to issue FPU instructions directly, which > > might have altered the ABI, or maybe the alternate libraries were > > just faster. > > > > > > Similarly Microsoft and memory models. Various switches changed the > > size of a pointer and implied a need for different libraries. > > > > > > Current gcc and ppc floating point sizes I believe is a contemporary > > example. > > Even sometimes the affect that -pthread has -- chosing different > > libraries (unnecessary > > mess imho, but one that seems to persist on HP-UX.) > > > > > > It's kind of an ugly area, to offer too many choices, produces > > confusion, but the limited important goal of increasing 64bit > > integer testing seems worth going here. And increasing userthread > > testing. People have expressed an interest in higher precision > > floating point, esp. 80 bit on x86, so this maybe a good way to > > proceed there also. > > You could also imagine flags -SSE, -SSE2, etc. > > > > > > It is important to only support a very limited number of these flags. > > Large cross products are not fun to worry about and test. > > But for example, with SSE, that nets you several targets all at > > once, without per-target work. > > > > (I lose the term "SSE" loosely. Could be I mean "SSE2" or 3 -- > > whatever is sufficient to largely replace the stack based x87.) > > > > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 16 11:39:09 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 11:39:09 +0100 Subject: [M3devel] release status [was something else] In-Reply-To: References: , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu> Message-ID: <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Quoting Jay K : > Release status as far as I know: > > - cvsupd hang > Reported over a year ago. Needs investigation. > Can anyone confirm it with a recent version? > And, as Tony asked, with user threads? If we can reproduce this, we should fix it; if not, I'd say this isn't relevant for the release. We can try to use a cvsupd from the release branch on birch serving the cm3 repository on a different port. Let's see if that works. Maybe tonight. > - NT386 has no 64bit longint in release branch. Partly the point of > this question. I thought Randy was going to do some more testing. If the m3back LONGINT changes don't break anything else, I'd be for just merging them to the branch and see if the builds and tests succeed. Of course testing with some other real applications would be great, but we must not delay endlessly for that. > - Should maybe improve build/packaging to get NT386-VC80.msi, > NT386-VC90.msi etc. (not the same as "target/abi alteration via > command line" expressed below, though that is a possibility, but > separate clean builds for now and/or separate CVS checkouts); really > more of a Hudson task now, to have multiple build environments > installed and used, I think I already updated the .msi building to > append the tag. Olaf? I'd rather not touch the NT386 setup on our virtual machine; perhaps you could provide another machine to run the Hudson jobs with a different Microsoft tools setup? > - I believe there is "new" divergence in m3front between head and > release -- whatever came along with the TInt changes. Is this relevant for the release? Or can we just ignore it? > Otherwise I'd have to look through trac. That's always a good idea. > There's maybe some stuff not merged from release to head, but that's > ok for release. > > (I need to check which branch Randy fixed the examples in.) > > I need to re-test the .msis. We still need to add them to the WWW export/display, don't we? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 16 12:21:36 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 11:21:36 +0000 Subject: [M3devel] release status [was something else] In-Reply-To: <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: Responding to just parts: > > - NT386 has no 64bit longint in release branch. Partly the point of > > this question. > > I thought Randy was going to do some more testing. If the m3back LONGINT > changes don't break anything else, I'd be for just merging them to the It is a big change. It is dependent on TInt/TWord changes, though that is flexible. I could copy and rename them "M3BackInt". Or we could take them -- they go with m3front changes. Or we could even do a little experiment: The reason I use TInt/TWord is to get 64bit integers portably to 32bit hosts. Previously INTEGER/Word was used. I could try using LONGINT/Long instead. However it is still a large change. There are some other unrelated changes, nothing too significant on its own (I always say), but it adds up: It isn't messed up, but it is maybe "beyond recognition" compared to the previous. (as was recently suggested) some renaming for, I claim, clarity some small optimizations (e.g. using shorter encodings sometimes, like a byte instead of a dword for constants in instructions, or equivalent shorter constructs like or reg,-1 instead of mov reg,-1), atomics support, actually in very good shape now, but could use more testing and optimization "rewrite" of insert/extract to no longer use the tables, I've always been bothered by those tables "rewrite" of set_singleton/set_member to be *much* smaller/faster set_singleton is just the bts instruction, instead of a function call set_member is just bt instruction plus carry flag extraction, instead of a function call cleanup of the way epilogues are sometimes "patched in" Only save/restore non-volatile registers that are actually used. Probably other stuff I'm forgetting. There's also small related changes in m3objfile: support for outputing more than 4 bytes at a time, support for "backing up". Some extra commenting. Some formating consistency (then always at end of line, else always on its own line). "Namespace flattening" (from foo import a,b,c). I need to test 64bit subrange checking. The code is a little fishy in that it deals with single registers. It is an optimization and could probably just be pessimized. Or maybe easy to fix. There's also the question as to if the TInt use is all subtely wrong as Tony seems to say. I don't understand yet. It *seems* to work well and it didn't previously pay much attention to types, it just used INTEGER everywhere. The changes are overall large. Diff the two trees, just in m3back. I actually think my earlier question might be the way to go to dramatically increase testing/coverage/confidence -- cm3 -test64 appends "_TEST64" to the build dir name (maybe maybe not the target name) and sets WORD_SIZE = "64" and BITSIZE(INTEGER) and ADDRESS to 64. I'd even consider something "simpler" where I actually create another complete target, I386_NT_TEST64, including the various entries in m3makefiles, Target.i3, etc. Maybe I can just try this locally with a one line change in Target.m3 though. I'll try that first. > branch and see if the builds and tests succeed. Of course testing with > some other real applications would be great, but we must not delay > endlessly for that. I have to accept releasing with 32bit LONGINT on NT386, due to the large overall change. Hopefully we arrange for more frequent releases somehow. > > - Should maybe improve build/packaging to get NT386-VC80.msi, > > I'd rather not touch the NT386 setup on our virtual machine; I probably won't leave a machine running 24/7. Can I just run the automation manually? Recall Cygwin sshd didn't work adequately at the time. Well, yes, I can always run whatever automation. Somewhat it is a matter of principle of going through the more official more automated process vs. a less official, more error prone, less trusted manual process. Installing additional toolsets on the VM really should work ok, without breaking the existing. Can you try? Maybe this: You create one .msi using the existing process. We'll make sure it is stamped "-VC90" or whatnot. I'll build a whole bunch of others, and they be available as "alternates", and we'll exclude the one corresponding to yours? You provide e.g. cm3-5.8-I386_NT-VC80.msi and I'll provide cm3-5.8-I386_NT-VC20.msi cm3-5.8-I386_NT-VC40.msi cm3-5.8-I386_NT-VC41.msi cm3-5.8-I386_NT-VC42.msi cm3-5.8-I386_NT-VC50.msi cm3-5.8-I386_NT-VC60.msi cm3-5.8-I386_NT-VC70.msi cm3-5.8-I386_NT-VC71.msi cm3-5.8-I386_NT-VC90.msi I realize some of these are quite old and out of use, but it's pretty simple to produce them all. > > - I believe there is "new" divergence in m3front between head and > Is this relevant for the release? Or can we just ignore it? I'd have to look, or Tony should say. To some extent it is tied with if we take NT386 64bit longint. > msi > We still need to add them to the WWW export/display, don't we? Definitely. Sorry of this isn't edited well..took a while already... - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 16 13:32:49 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 12:32:49 +0000 Subject: [M3devel] cvsup Message-ID: I can reproduce the cvsup problem. An important point to consider is that cvsupd forks a server? Maybe that messes up initialization? I'll dig further. I think these steps are complete. I'll test them on a second clean system. You don't need any real data. jay at xlin2:~$ cat cvsupd.cfg *default tag=. *default host=xlin2. *default prefix=/home/jay *default base=/home/jay/var/db *default release=cvs delete use-rel-suffix compress src-all mkdir -p $HOME/var/db mkdir -p $HOME/data/cvsupd pkill cvsupd # cleanup any previous You don't really need any data. cvsup/cvsupd will issue an error in the "working" case, and hang in the non-working case. run the server: jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd 2010.03.16 05:24:25 PDT [22555]: CVSup server started 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 2010.03.16 05:24:25 PDT [22555]: Ready to service requests run the client: /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg Parsing supfile "/home/jay/cvsupd.cfg" Connecting to xlin2. Connected to xlin2. Server software version: 2010-03-05 Negotiating file attribute support Exchanging collection information Server message: Unknown collection "src-all" Establishing multiplexed-mode data connection Running Skipping collection src-all/cvs Shutting down connection to server Finished successfully it is able to talk to the server, and then fails. Ok -- at least it talked to the server. The server then exits too. I think that is correct (read the usage on the -C option). Then try with -C. The bug says -C 2. Let's use -C 1. They behave the same for our purposes. Just add -C 1 to the above server. Run the same client. The client hangs -- it never connects to the server. I reproduced this on LINUXLIBC6 and PPC_LINUX. Maybe more soon. I'm just rebuilding first. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 16 13:38:30 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 13:38:30 +0100 Subject: [M3devel] release status [was something else] In-Reply-To: References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: <20100316133830.8pcec289kcsc0sgk@mail.elegosoft.com> Quoting Jay K : > It is a big change. [...] > I actually think my earlier question might be the way to go to dramatically > increase testing/coverage/confidence -- cm3 -test64 appends "_TEST64" > to the build dir name (maybe maybe not the target name) and sets > WORD_SIZE = "64" and BITSIZE(INTEGER) and ADDRESS to 64. > > I'd even consider something "simpler" where I actually create > another complete > target, I386_NT_TEST64, including the various entries in > m3makefiles, Target.i3, etc. > > Maybe I can just try this locally with a one line change in Target.m3 though. > > I'll try that first. That would be good. [...] > I have to accept releasing with 32bit LONGINT on NT386, due to the large > overall change. > > Hopefully we arrange for more frequent releases somehow. So you think we should not risk the merge and release without the m3back changes now? >> > - Should maybe improve build/packaging to get NT386-VC80.msi, >> >> I'd rather not touch the NT386 setup on our virtual machine; > > I probably won't leave a machine running 24/7. > > Can I just run the automation manually? You can copy and run all the Hudson jobs manually. I'd suggest a parametric setup for the differences though. > Recall Cygwin sshd didn't work adequately at the time. > Well, yes, I can always run whatever automation. Somewhat > it is a matter of principle of going through the more official > more automated > process vs. a less official, more error prone, less trusted > manual process. > Installing additional toolsets on the VM really should work ok, > without breaking > the existing. Can you try? I cannot really do that without GUI access; and I'm afraid there are currently no ressources to setup all those tools (Kay's busy with other work). > Maybe this: You create one .msi using the existing process. We'll make > sure it is stamped "-VC90" or whatnot. > > I'll build a whole bunch of others, and they be available as "alternates", > and we'll exclude the one corresponding to yours? > > You provide e.g. cm3-5.8-I386_NT-VC80.msi and I'll provide > > cm3-5.8-I386_NT-VC20.msi > cm3-5.8-I386_NT-VC40.msi > cm3-5.8-I386_NT-VC41.msi > cm3-5.8-I386_NT-VC42.msi > cm3-5.8-I386_NT-VC50.msi > cm3-5.8-I386_NT-VC60.msi > cm3-5.8-I386_NT-VC70.msi > cm3-5.8-I386_NT-VC71.msi > > cm3-5.8-I386_NT-VC90.msi Do we really need all those? We would not only need the msi installers, but the other packages would have to be compiled and linked with the different tools, too, or they'd be incompatible, wouldn't they? > I realize some of these are quite old and out of use, but it's > pretty simple to produce them all. Of course you can contribute whatever installation archives you can create. I'd rather have a defined set of target platforms and build procedures for the official release though. So I think we should limit our set of supported tool sets. What would you suggest that is most likely to be really useful? > > > - I believe there is "new" divergence in m3front between head and > > Is this relevant for the release? Or can we just ignore it? > > I'd have to look, or Tony should say. > > To some extent it is tied with if we take NT386 64bit longint. > > We still need to add them to the WWW export/display, don't we? > > Definitely. I'll do that. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From hosking at cs.purdue.edu Tue Mar 16 16:07:47 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 11:07:47 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: References: , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu> Message-ID: On 15 Mar 2010, at 21:41, Jay K wrote: > Release status as far as I know: > > - cvsupd hang > Reported over a year ago. Needs investigation. > Can anyone confirm it with a recent version? > And, as Tony asked, with user threads? If anyone gets a snapshot please also make sure to get a backtrace for all the threads. > - NT386 has no 64bit longint in release branch. Partly the point of this question. > > > - Should maybe improve build/packaging to get NT386-VC80.msi, NT386-VC90.msi etc. (not the same as "target/abi alteration via command line" expressed below, though that is a possibility, but separate clean builds for now and/or separate CVS checkouts); really more of a Hudson task now, to have multiple build environments installed and used, I think I already updated the .msi building to append the tag. Olaf? > > > - I believe there is "new" divergence in m3front between head and release -- whatever came along with the TInt changes. Those changes will need bringing over from head to release if Jay's m3back 64-bit support for NT386 are also brought over. I believe there are a few dependencies. > Otherwise I'd have to look through trac. > > > There's maybe some stuff not merged from release to head, but that's ok for release. > (I need to check which branch Randy fixed the examples in.) > I need to re-test the .msis. > > > - Jay > > > > From: hosking at cs.purdue.edu > > To: jay.krell at cornell.edu > > Subject: Re: [M3devel] arbitrary use of LONGINT for testing? target/ABI alteration/generation via command line? > > Date: Mon, 15 Mar 2010 20:00:27 -0500 > > CC: m3devel at elegosoft.com > > > > Let's not right now. We need to get the release out. I suggest a > > freeze on compiler changes for now. Bug fixes only! > > > > Sent from my iPhone > > > > On Mar 15, 2010, at 7:54 PM, Jay K wrote: > > > > > Tony et. al. I'm wondering/fishing if you any ideas, like a cm3 > > > command line switch, that might be used to make INTEGER 64bits to > > > increase coverage/testing of 64bit integer support. > > > > > > > > > The problem is *at least* interfacing with external code/files, if > > > not breaking everything. > > > Maybe also a problem around sizeof(INTEGER) vs. sizeof(ADDRESS). > > > Probably such a setting would grow ADDRESS too. > > > > > > > > > Like, maybe stuff like m3core/unix, m3core/win32 should never use > > > plain INTEGER or LONGINT, but "BITS FOR", to allow this to work? Or > > > maybe <* EXTERNAL *> would be the hint? > > > No, <* EXTERNAL *> isn't adequate, due to the case of callbacks. > > > > > > > > > I'd like to somehow, without too much effort, significantly increase > > > testing of LONGINT. > > > > > > > > > Maybe a wierdo platform NT386_64 that does this just for test > > > purposes? > > > I might try that, if "BITS FOR" is enough to actually let it work > > > (interface with C). > > > (or generally, cm3 -test64 would append "_TEST64" to target name, > > > so we could have LINUXLIBC6_TEST64, I386_DARWIN_TEST64, etc.) > > > > > > > > > I can do some initial experiments along this -test64 line, see if it > > > completely blows up or not. > > > A few minutes thought suggests it should work easily and provide > > > value. > > > > > > > > > (There's probably something similar to try around a 16 bit CHAR. vs. > > > BITS 8 FOR CHAR, but I'm not going there..) > > > > > > > > > Such command line driven target/ABI alterations seem like a somewhat > > > good idea. > > > e.g. cm3 -userthreads would append _USERTHREADS. > > > cm3 -extended80 would use an 80 or 96 bit extended on x86 (96 bits > > > for alignment). > > > -extended128 on ppc > > > etc. > > > > > > > > > cm3 -msvc80 could probe around in enviroment for a Visual C++ 8.0 > > > compiler/linker/header/libraries and append "_MSVC80" to target, > > > else fail. > > > > > > > > > I'll just try -test64 for now. > > > The rest aren't as immediately useful. > > > -userthreads would be my next "favorate". > > > > > > Hm. Instead of -test64, how about -64. > > > If integer is 32 bits, change it to 64 and append _64 (and address). > > > If integer is already 64bits, do nothing. > > > > > > > > > This is quite different than gcc's -m64 though. > > > Maybe not good that way. > > > > > > > > > You can see parallels in several older C compilers. > > > > > > Metrowerks for Mac/68K let you chose integer size and either double > > > or extended size (I forget which). > > > That gave you a cross product set of ABIs, each with their own > > > libraries. > > > > > > > > > Similarly you could chose to issue FPU instructions directly, which > > > might have altered the ABI, or maybe the alternate libraries were > > > just faster. > > > > > > > > > Similarly Microsoft and memory models. Various switches changed the > > > size of a pointer and implied a need for different libraries. > > > > > > > > > Current gcc and ppc floating point sizes I believe is a contemporary > > > example. > > > Even sometimes the affect that -pthread has -- chosing different > > > libraries (unnecessary > > > mess imho, but one that seems to persist on HP-UX.) > > > > > > > > > It's kind of an ugly area, to offer too many choices, produces > > > confusion, but the limited important goal of increasing 64bit > > > integer testing seems worth going here. And increasing userthread > > > testing. People have expressed an interest in higher precision > > > floating point, esp. 80 bit on x86, so this maybe a good way to > > > proceed there also. > > > You could also imagine flags -SSE, -SSE2, etc. > > > > > > > > > It is important to only support a very limited number of these flags. > > > Large cross products are not fun to worry about and test. > > > But for example, with SSE, that nets you several targets all at > > > once, without per-target work. > > > > > > (I lose the term "SSE" loosely. Could be I mean "SSE2" or 3 -- > > > whatever is sufficient to largely replace the stack based x87.) > > > > > > > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 16:21:41 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 11:21:41 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: Ah, yes, one issue about bringing over m3front changes is that it also includes the atomics support. I don't think we want to do this in this release. So, this argues that we hold off on releasing the NT386 64-bit LONGINT support for now. Thoughts? On 16 Mar 2010, at 07:21, Jay K wrote: > Responding to just parts: > > > > > - NT386 has no 64bit longint in release branch. Partly the point of > > > this question. > > > > I thought Randy was going to do some more testing. If the m3back LONGINT > > changes don't break anything else, I'd be for just merging them to the > > > It is a big change. > It is dependent on TInt/TWord changes, though that is flexible. > I could copy and rename them "M3BackInt". > Or we could take them -- they go with m3front changes. > > > Or we could even do a little experiment: The reason I use TInt/TWord > is to get 64bit integers portably to 32bit hosts. Previously INTEGER/Word was used. > I could try using LONGINT/Long instead. > > > However it is still a large change. There are some other unrelated changes, nothing > too significant on its own (I always say), but it adds up: > It isn't messed up, but it is maybe "beyond recognition" compared to the previous. > (as was recently suggested) > > some renaming for, I claim, clarity > > some small optimizations (e.g. using shorter encodings sometimes, like > a byte instead of a dword for constants in instructions, or > equivalent shorter constructs like or reg,-1 instead of mov reg,-1), > > atomics support, actually in very good shape now, but could use more testing and optimization > > "rewrite" of insert/extract to no longer use the tables, I've always > been bothered by those tables > > "rewrite" of set_singleton/set_member to be *much* smaller/faster > set_singleton is just the bts instruction, instead of a function call > set_member is just bt instruction plus carry flag extraction, instead of a function call > > cleanup of the way epilogues are sometimes "patched in" > > Only save/restore non-volatile registers that are actually used. > > Probably other stuff I'm forgetting. > > There's also small related changes in m3objfile: support for outputing more than 4 > bytes at a time, support for "backing up". > Some extra commenting. > Some formating consistency (then always at end of line, else always on its own line). > "Namespace flattening" (from foo import a,b,c). > > I need to test 64bit subrange checking. > The code is a little fishy in that it deals with single registers. > It is an optimization and could probably just be pessimized. > Or maybe easy to fix. > > > There's also the question as to if the TInt use is all subtely wrong as Tony seems to say. > I don't understand yet. > It *seems* to work well and it didn't previously pay much attention to types, it > just used INTEGER everywhere. > > > The changes are overall large. > Diff the two trees, just in m3back. > > > I actually think my earlier question might be the way to go to dramatically > increase testing/coverage/confidence -- cm3 -test64 appends "_TEST64" > to the build dir name (maybe maybe not the target name) and sets > WORD_SIZE = "64" and BITSIZE(INTEGER) and ADDRESS to 64. > I'd even consider something "simpler" where I actually create another complete > target, I386_NT_TEST64, including the various entries in m3makefiles, Target.i3, etc. > > > Maybe I can just try this locally with a one line change in Target.m3 though. > I'll try that first. > > > > branch and see if the builds and tests succeed. Of course testing with > > some other real applications would be great, but we must not delay > > endlessly for that. > > > I have to accept releasing with 32bit LONGINT on NT386, due to the large > overall change. > Hopefully we arrange for more frequent releases somehow. > > > > > > - Should maybe improve build/packaging to get NT386-VC80.msi, > > > > I'd rather not touch the NT386 setup on our virtual machine; > > > I probably won't leave a machine running 24/7. > Can I just run the automation manually? > Recall Cygwin sshd didn't work adequately at the time. > Well, yes, I can always run whatever automation. Somewhat > it is a matter of principle of going through the more official more automated > process vs. a less official, more error prone, less trusted manual process. > > > Installing additional toolsets on the VM really should work ok, without breaking > the existing. Can you try? > > > Maybe this: You create one .msi using the existing process. We'll make > sure it is stamped "-VC90" or whatnot. > I'll build a whole bunch of others, and they be available as "alternates", > and we'll exclude the one corresponding to yours? > You provide e.g. cm3-5.8-I386_NT-VC80.msi and I'll provide > cm3-5.8-I386_NT-VC20.msi > cm3-5.8-I386_NT-VC40.msi > cm3-5.8-I386_NT-VC41.msi > cm3-5.8-I386_NT-VC42.msi > cm3-5.8-I386_NT-VC50.msi > cm3-5.8-I386_NT-VC60.msi > cm3-5.8-I386_NT-VC70.msi > cm3-5.8-I386_NT-VC71.msi > > cm3-5.8-I386_NT-VC90.msi > > > I realize some of these are quite old and out of use, but it's pretty simple to produce > them all. > > > > > - I believe there is "new" divergence in m3front between head and > > Is this relevant for the release? Or can we just ignore it? > > I'd have to look, or Tony should say. > To some extent it is tied with if we take NT386 64bit longint. > > > > msi > > We still need to add them to the WWW export/display, don't we? > > > Definitely. > > > Sorry of this isn't edited well..took a while already... > > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 16:23:34 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 11:23:34 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: <20100316133830.8pcec289kcsc0sgk@mail.elegosoft.com> References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> <20100316133830.8pcec289kcsc0sgk@mail.elegosoft.com> Message-ID: <720A9819-E452-4CEB-A0AA-149B1FF6D88E@cs.purdue.edu> On 16 Mar 2010, at 08:38, Olaf Wagner wrote: > Quoting Jay K : > >> It is a big change. > [...] >> I actually think my earlier question might be the way to go to dramatically >> increase testing/coverage/confidence -- cm3 -test64 appends "_TEST64" >> to the build dir name (maybe maybe not the target name) and sets >> WORD_SIZE = "64" and BITSIZE(INTEGER) and ADDRESS to 64. >> >> I'd even consider something "simpler" where I actually create another complete >> target, I386_NT_TEST64, including the various entries in m3makefiles, Target.i3, etc. >> >> Maybe I can just try this locally with a one line change in Target.m3 though. >> >> I'll try that first. > That would be good. That has pervasive implications. I don't know that there are not large swathes of code that assume that BITSIZE(INTEGER)=BITSIZE(ADDRESS). > > [...] >> I have to accept releasing with 32bit LONGINT on NT386, due to the large >> overall change. Yes, I think that is the way we need to go -- defer 64-bit NT386 LONGINT for now. >> >> Hopefully we arrange for more frequent releases somehow. > > So you think we should not risk the merge and release without the > m3back changes now? Yes. > >>> > - Should maybe improve build/packaging to get NT386-VC80.msi, >>> >>> I'd rather not touch the NT386 setup on our virtual machine; >> >> I probably won't leave a machine running 24/7. >> >> Can I just run the automation manually? > > You can copy and run all the Hudson jobs manually. I'd suggest a > parametric setup for the differences though. > >> Recall Cygwin sshd didn't work adequately at the time. >> Well, yes, I can always run whatever automation. Somewhat >> it is a matter of principle of going through the more official more automated >> process vs. a less official, more error prone, less trusted manual process. > >> Installing additional toolsets on the VM really should work ok, without breaking >> the existing. Can you try? > > I cannot really do that without GUI access; and I'm afraid there are > currently no ressources to setup all those tools (Kay's busy with other > work). > >> Maybe this: You create one .msi using the existing process. We'll make >> sure it is stamped "-VC90" or whatnot. >> >> I'll build a whole bunch of others, and they be available as "alternates", >> and we'll exclude the one corresponding to yours? >> >> You provide e.g. cm3-5.8-I386_NT-VC80.msi and I'll provide >> >> cm3-5.8-I386_NT-VC20.msi >> cm3-5.8-I386_NT-VC40.msi >> cm3-5.8-I386_NT-VC41.msi >> cm3-5.8-I386_NT-VC42.msi >> cm3-5.8-I386_NT-VC50.msi >> cm3-5.8-I386_NT-VC60.msi >> cm3-5.8-I386_NT-VC70.msi >> cm3-5.8-I386_NT-VC71.msi >> >> cm3-5.8-I386_NT-VC90.msi > > Do we really need all those? We would not only need the msi installers, > but the other packages would have to be compiled and linked with the > different tools, too, or they'd be incompatible, wouldn't they? > >> I realize some of these are quite old and out of use, but it's pretty simple to produce them all. > > Of course you can contribute whatever installation archives you can create. > I'd rather have a defined set of target platforms and build procedures > for the official release though. So I think we should limit our > set of supported tool sets. What would you suggest that is most likely > to be really useful? > >> > > - I believe there is "new" divergence in m3front between head and >> > Is this relevant for the release? Or can we just ignore it? >> >> I'd have to look, or Tony should say. >> >> To some extent it is tied with if we take NT386 64bit longint. > >> > We still need to add them to the WWW export/display, don't we? >> >> Definitely. > > I'll do that. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > From wagner at elegosoft.com Tue Mar 16 16:28:42 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 16:28:42 +0100 Subject: [M3devel] release status [was something else] In-Reply-To: References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: <20100316162842.qzx2mmd7cc8kockc@mail.elegosoft.com> Quoting Tony Hosking : > Ah, yes, one issue about bringing over m3front changes is that it > also includes the atomics support. I don't think we want to do this > in this release. > So, this argues that we hold off on releasing the NT386 64-bit > LONGINT support for now. > > Thoughts? It's OK by me. You have convinced me that there are too many dependencies and implications. We should be able to start a new RC build in a few days then. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 16 16:33:29 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 15:33:29 +0000 Subject: [M3devel] cvsup Message-ID: Daniel thank you for the bug report. Thank you for suggesting strace. I used strace and compared working vs. non-working. And started added RTIO everywhere. You can also use cvsup -f to slightly simplify -- one fork instead of two. gdb set follow mode didn't seem to help. I almost have it nailed down. in CVSup, FSServer.m3, this code: FINALLY IF isChild THEN SigHandler.ShutDown(); <== ELSE SigHandler.Unblock(); END; END; which runs fairly early, never returns in the child. It ends up here: PROCEDURE ChangeState(d: Dispatcher; state: State) = (* Ask the dispatcher thread to change to a new state, and wait until it has made the change. *) BEGIN LOCK d.mu DO d.desiredState := state; IF d.state # d.desiredState THEN IF d.state = State.Running THEN (* Send dummy signal 0 to wake up the dispatcher. *) Catch(0); ELSE Thread.Signal(d.changeState); END; WHILE d.state # d.desiredState DO Thread.Wait(d.mu, d.stateChanged); <== this never returns END; END; END; END ChangeState; It's a bit wierd to be mixing fork() and Modula-3 Thread? Or maybe it is ok? See, they are asking another process, from the fork() point of view, to change the state. It does so, but the write is private. Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? sbrk is maybe shared? mmap(anon|private) is not. Right now I have #ifndef apple sbrk #else mmap(anon|shared) #endif and it gets further. Hit an assertion failure in pthread. I'll try again. Cleanup all my RTIO. Maybe this notion of using fork() is just not supportable? In either case...you could paint it as an m3core problem, but ?it won't affect much code?. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: cvsup Date: Tue, 16 Mar 2010 12:32:49 +0000 I can reproduce the cvsup problem. An important point to consider is that cvsupd forks a server? Maybe that messes up initialization? I'll dig further. I think these steps are complete. I'll test them on a second clean system. You don't need any real data. jay at xlin2:~$ cat cvsupd.cfg *default tag=. *default host=xlin2. *default prefix=/home/jay *default base=/home/jay/var/db *default release=cvs delete use-rel-suffix compress src-all mkdir -p $HOME/var/db mkdir -p $HOME/data/cvsupd pkill cvsupd # cleanup any previous You don't really need any data. cvsup/cvsupd will issue an error in the "working" case, and hang in the non-working case. run the server: jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd 2010.03.16 05:24:25 PDT [22555]: CVSup server started 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 2010.03.16 05:24:25 PDT [22555]: Ready to service requests run the client: /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg Parsing supfile "/home/jay/cvsupd.cfg" Connecting to xlin2. Connected to xlin2. Server software version: 2010-03-05 Negotiating file attribute support Exchanging collection information Server message: Unknown collection "src-all" Establishing multiplexed-mode data connection Running Skipping collection src-all/cvs Shutting down connection to server Finished successfully it is able to talk to the server, and then fails. Ok -- at least it talked to the server. The server then exits too. I think that is correct (read the usage on the -C option). Then try with -C. The bug says -C 2. Let's use -C 1. They behave the same for our purposes. Just add -C 1 to the above server. Run the same client. The client hangs -- it never connects to the server. I reproduced this on LINUXLIBC6 and PPC_LINUX. Maybe more soon. I'm just rebuilding first. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 16:43:52 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 11:43:52 -0400 Subject: [M3devel] cvsup In-Reply-To: References: Message-ID: <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> mmap is strongly preferred over sbrk. I'd need to understand the control flow here in and across the processes, but yes, generally, mixing pthreads with fork may require significant care. On 16 Mar 2010, at 11:33, Jay K wrote: > Daniel thank you for the bug report. > Thank you for suggesting strace. I used strace > and compared working vs. non-working. > And started added RTIO everywhere. > You can also use cvsup -f to slightly simplify -- one fork instead of two. > gdb set follow mode didn't seem to help. > > > I almost have it nailed down. > > > in CVSup, FSServer.m3, this code: > > FINALLY > IF isChild THEN > SigHandler.ShutDown(); <== > ELSE > SigHandler.Unblock(); > END; > END; > > > which runs fairly early, never returns in the child. > > > It ends up here: > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > (* Ask the dispatcher thread to change to a new state, and wait until > it has made the change. *) > BEGIN > LOCK d.mu DO > d.desiredState := state; > IF d.state # d.desiredState THEN > IF d.state = State.Running THEN > (* Send dummy signal 0 to wake up the dispatcher. *) > Catch(0); > ELSE > Thread.Signal(d.changeState); > END; > WHILE d.state # d.desiredState DO > Thread.Wait(d.mu, d.stateChanged); <== this never returns > END; > END; > END; > END ChangeState; > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > Or maybe it is ok? > > > See, they are asking another process, from the fork() point of view, to change the state. > It does so, but the write is private. > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > sbrk is maybe shared? mmap(anon|private) is not. > > Right now I have > #ifndef apple > sbrk > #else > mmap(anon|shared) > #endif > > > and it gets further. > Hit an assertion failure in pthread. > I'll try again. > Cleanup all my RTIO. > > > Maybe this notion of using fork() is just not supportable? > > > In either case...you could paint it as an m3core problem, but > ?it won't affect much code?. > > > - Jay > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Subject: cvsup > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > I can reproduce the cvsup problem. > An important point to consider is that cvsupd forks a server? > Maybe that messes up initialization? > I'll dig further. > > > I think these steps are complete. > I'll test them on a second clean system. > You don't need any real data. > > > jay at xlin2:~$ cat cvsupd.cfg > *default tag=. > *default host=xlin2. > *default prefix=/home/jay > *default base=/home/jay/var/db > *default release=cvs delete use-rel-suffix compress > > src-all > > > mkdir -p $HOME/var/db > mkdir -p $HOME/data/cvsupd > pkill cvsupd # cleanup any previous > > > You don't really need any data. > cvsup/cvsupd will issue an error in the "working" case, and > hang in the non-working case. > > > > > run the server: > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > run the client: > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > Parsing supfile "/home/jay/cvsupd.cfg" > Connecting to xlin2. > Connected to xlin2. > Server software version: 2010-03-05 > Negotiating file attribute support > Exchanging collection information > Server message: Unknown collection "src-all" > Establishing multiplexed-mode data connection > Running > Skipping collection src-all/cvs > Shutting down connection to server > Finished successfully > > it is able to talk to the server, and then fails. > Ok -- at least it talked to the server. > > The server then exits too. > I think that is correct (read the usage on the -C option). > > > Then try with -C. > The bug says -C 2. Let's use -C 1. > They behave the same for our purposes. > > > Just add -C 1 to the above server. > Run the same client. > The client hangs -- it never connects to the server. > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > Maybe more soon. > I'm just rebuilding first. > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 16 16:51:42 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 15:51:42 +0000 Subject: [M3devel] release status [was something else] In-Reply-To: <20100316162842.qzx2mmd7cc8kockc@mail.elegosoft.com> References: , , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , , , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com>, , , <20100316162842.qzx2mmd7cc8kockc@mail.elegosoft.com> Message-ID: The "experiment" would also probably have to grow ADDRESS. And then for interop I'd have to say, like HANDLE = BITS 32 FOR ADDRESS, if that is allowed. And then..it gets confused..where would I get "32" from? Compiler would have to truncate/extend pointers. Not sure it is willing to. Another good theoretical route is, indeed widen all the pointers, and then #ifdef the C code to take UINT64 and cast to pointers..and convert structs back/forth..but for Win32 we generally don't have such C code as we do for Posix. Darn. I'll think about it more later but maybe it doesn't really work out. - Jay > Date: Tue, 16 Mar 2010 16:28:42 +0100 > From: wagner at elegosoft.com > To: hosking at cs.purdue.edu > CC: jay.krell at cornell.edu; m3devel at elegosoft.com > Subject: Re: [M3devel] release status [was something else] > > Quoting Tony Hosking : > > > Ah, yes, one issue about bringing over m3front changes is that it > > also includes the atomics support. I don't think we want to do this > > in this release. > > So, this argues that we hold off on releasing the NT386 64-bit > > LONGINT support for now. > > > > Thoughts? > > It's OK by me. You have convinced me that there are too many dependencies > and implications. > > We should be able to start a new RC build in a few days then. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 16 17:10:15 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 16:10:15 +0000 Subject: [M3devel] cvsup In-Reply-To: <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> References: , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> Message-ID: I could have sworn I had not seen hanging with sbrk or map_shared, but I can't get that now. I'll dig a bit more..maybe not today. To be clear, this isn't even the usual fork+exec. This is fork + do work + exit. We can probably fix it by having it create a Modula-3 thread. The first fork (daemonize) is probably fine. It is the per-client one that I think is a problem. I'd like to find a theory as to why it ever worked, maybe see it work, and then fix it differently. Maybe sbrk + user threads? (Can anyone claim to have seen cvsup server work with kernel threads?) - Jay From: hosking at cs.purdue.edu Date: Tue, 16 Mar 2010 11:43:52 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup mmap is strongly preferred over sbrk. I'd need to understand the control flow here in and across the processes, but yes, generally, mixing pthreads with fork may require significant care. On 16 Mar 2010, at 11:33, Jay K wrote: Daniel thank you for the bug report. Thank you for suggesting strace. I used strace and compared working vs. non-working. And started added RTIO everywhere. You can also use cvsup -f to slightly simplify -- one fork instead of two. gdb set follow mode didn't seem to help. I almost have it nailed down. in CVSup, FSServer.m3, this code: FINALLY IF isChild THEN SigHandler.ShutDown(); <== ELSE SigHandler.Unblock(); END; END; which runs fairly early, never returns in the child. It ends up here: PROCEDURE ChangeState(d: Dispatcher; state: State) = (* Ask the dispatcher thread to change to a new state, and wait until it has made the change. *) BEGIN LOCK d.mu DO d.desiredState := state; IF d.state # d.desiredState THEN IF d.state = State.Running THEN (* Send dummy signal 0 to wake up the dispatcher. *) Catch(0); ELSE Thread.Signal(d.changeState); END; WHILE d.state # d.desiredState DO Thread.Wait(d.mu, d.stateChanged); <== this never returns END; END; END; END ChangeState; It's a bit wierd to be mixing fork() and Modula-3 Thread? Or maybe it is ok? See, they are asking another process, from the fork() point of view, to change the state. It does so, but the write is private. Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? sbrk is maybe shared? mmap(anon|private) is not. Right now I have #ifndef apple sbrk #else mmap(anon|shared) #endif and it gets further. Hit an assertion failure in pthread. I'll try again. Cleanup all my RTIO. Maybe this notion of using fork() is just not supportable? In either case...you could paint it as an m3core problem, but ?it won't affect much code?. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: cvsup Date: Tue, 16 Mar 2010 12:32:49 +0000 I can reproduce the cvsup problem. An important point to consider is that cvsupd forks a server? Maybe that messes up initialization? I'll dig further. I think these steps are complete. I'll test them on a second clean system. You don't need any real data. jay at xlin2:~$ cat cvsupd.cfg *default tag=. *default host=xlin2. *default prefix=/home/jay *default base=/home/jay/var/db *default release=cvs delete use-rel-suffix compress src-all mkdir -p $HOME/var/db mkdir -p $HOME/data/cvsupd pkill cvsupd # cleanup any previous You don't really need any data. cvsup/cvsupd will issue an error in the "working" case, and hang in the non-working case. run the server: jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd 2010.03.16 05:24:25 PDT [22555]: CVSup server started 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 2010.03.16 05:24:25 PDT [22555]: Ready to service requests run the client: /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg Parsing supfile "/home/jay/cvsupd.cfg" Connecting to xlin2. Connected to xlin2. Server software version: 2010-03-05 Negotiating file attribute support Exchanging collection information Server message: Unknown collection "src-all" Establishing multiplexed-mode data connection Running Skipping collection src-all/cvs Shutting down connection to server Finished successfully it is able to talk to the server, and then fails. Ok -- at least it talked to the server. The server then exits too. I think that is correct (read the usage on the -C option). Then try with -C. The bug says -C 2. Let's use -C 1. They behave the same for our purposes. Just add -C 1 to the above server. Run the same client. The client hangs -- it never connects to the server. I reproduced this on LINUXLIBC6 and PPC_LINUX. Maybe more soon. I'm just rebuilding first. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 16 17:35:40 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 17:35:40 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> Message-ID: <20100316173540.7cgvahkys44ggw4o@mail.elegosoft.com> Quoting Jay K : > I could have sworn I had not seen hanging with sbrk or map_shared, > but I can't get that now. > > I'll dig a bit more..maybe not today. > > To be clear, this isn't even the usual fork+exec. This is fork + do > work + exit. Well, that's how processes on Unix are usually used for scaling. > We can probably fix it by having it create a Modula-3 thread. Hm. That might work, but wouldn't really be the intention. There should be several server processes each with several threads. > The first fork (daemonize) is probably fine. It is the per-client > one that I think is a problem. > > I'd like to find a theory as to why it ever worked, maybe see it > work, and then fix it differently. > > Maybe sbrk + user threads? > > (Can anyone claim to have seen cvsup server work with kernel threads?) No. I think we always used older binaries at Elego. We should be able to make it work though ;-) Olaf > - Jay > > > > From: hosking at cs.purdue.edu > Date: Tue, 16 Mar 2010 11:43:52 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > mmap is strongly preferred over sbrk. > > > I'd need to understand the control flow here in and across the > processes, but yes, generally, mixing pthreads with fork may require > significant care. > > > > On 16 Mar 2010, at 11:33, Jay K wrote: > > Daniel thank you for the bug report. > Thank you for suggesting strace. I used strace > and compared working vs. non-working. > And started added RTIO everywhere. > You can also use cvsup -f to slightly simplify -- one fork instead of two. > gdb set follow mode didn't seem to help. > > > I almost have it nailed down. > > > in CVSup, FSServer.m3, this code: > > FINALLY > IF isChild THEN > SigHandler.ShutDown(); <== > ELSE > SigHandler.Unblock(); > END; > END; > > > which runs fairly early, never returns in the child. > > > It ends up here: > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > (* Ask the dispatcher thread to change to a new state, and wait until > it has made the change. *) > BEGIN > LOCK d.mu DO > d.desiredState := state; > IF d.state # d.desiredState THEN > IF d.state = State.Running THEN > (* Send dummy signal 0 to wake up the dispatcher. *) > Catch(0); > ELSE > Thread.Signal(d.changeState); > END; > WHILE d.state # d.desiredState DO > Thread.Wait(d.mu, d.stateChanged); <== this never returns > END; > END; > END; > END ChangeState; > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > Or maybe it is ok? > > > See, they are asking another process, from the fork() point of view, > to change the state. > It does so, but the write is private. > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > sbrk is maybe shared? mmap(anon|private) is not. > > Right now I have > #ifndef apple > sbrk > #else > mmap(anon|shared) > #endif > > > and it gets further. > Hit an assertion failure in pthread. > I'll try again. > Cleanup all my RTIO. > > > Maybe this notion of using fork() is just not supportable? > > > In either case...you could paint it as an m3core problem, but > ?it won't affect much code?. > > > - Jay > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Subject: cvsup > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > I can reproduce the cvsup problem. > An important point to consider is that cvsupd forks a server? > Maybe that messes up initialization? > I'll dig further. > > > I think these steps are complete. > I'll test them on a second clean system. > You don't need any real data. > > > jay at xlin2:~$ cat cvsupd.cfg > *default tag=. > *default host=xlin2. > *default prefix=/home/jay > *default base=/home/jay/var/db > *default release=cvs delete use-rel-suffix compress > > src-all > > > mkdir -p $HOME/var/db > mkdir -p $HOME/data/cvsupd > pkill cvsupd # cleanup any previous > > > You don't really need any data. > cvsup/cvsupd will issue an error in the "working" case, and > hang in the non-working case. > > > > > run the server: > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > run the client: > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > Parsing supfile "/home/jay/cvsupd.cfg" > Connecting to xlin2. > Connected to xlin2. > Server software version: 2010-03-05 > Negotiating file attribute support > Exchanging collection information > Server message: Unknown collection "src-all" > Establishing multiplexed-mode data connection > Running > Skipping collection src-all/cvs > Shutting down connection to server > Finished successfully > > it is able to talk to the server, and then fails. > Ok -- at least it talked to the server. > > The server then exits too. > I think that is correct (read the usage on the -C option). > > > Then try with -C. > The bug says -C 2. Let's use -C 1. > They behave the same for our purposes. > > > Just add -C 1 to the above server. > Run the same client. > The client hangs -- it never connects to the server. > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > Maybe more soon. > I'm just rebuilding first. > > > - Jay > > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 16 17:38:00 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 16:38:00 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, Message-ID: User threads is apparently sufficient to let it work -- no change to sbrk vs. mmap. You end up with a wierd mix of shared and not shared data, right? I'll try to make it use pthreads and Modula-3 threads and not fork(). - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Tue, 16 Mar 2010 16:10:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup I could have sworn I had not seen hanging with sbrk or map_shared, but I can't get that now. I'll dig a bit more..maybe not today. To be clear, this isn't even the usual fork+exec. This is fork + do work + exit. We can probably fix it by having it create a Modula-3 thread. The first fork (daemonize) is probably fine. It is the per-client one that I think is a problem. I'd like to find a theory as to why it ever worked, maybe see it work, and then fix it differently. Maybe sbrk + user threads? (Can anyone claim to have seen cvsup server work with kernel threads?) - Jay From: hosking at cs.purdue.edu Date: Tue, 16 Mar 2010 11:43:52 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup mmap is strongly preferred over sbrk. I'd need to understand the control flow here in and across the processes, but yes, generally, mixing pthreads with fork may require significant care. On 16 Mar 2010, at 11:33, Jay K wrote: Daniel thank you for the bug report. Thank you for suggesting strace. I used strace and compared working vs. non-working. And started added RTIO everywhere. You can also use cvsup -f to slightly simplify -- one fork instead of two. gdb set follow mode didn't seem to help. I almost have it nailed down. in CVSup, FSServer.m3, this code: FINALLY IF isChild THEN SigHandler.ShutDown(); <== ELSE SigHandler.Unblock(); END; END; which runs fairly early, never returns in the child. It ends up here: PROCEDURE ChangeState(d: Dispatcher; state: State) = (* Ask the dispatcher thread to change to a new state, and wait until it has made the change. *) BEGIN LOCK d.mu DO d.desiredState := state; IF d.state # d.desiredState THEN IF d.state = State.Running THEN (* Send dummy signal 0 to wake up the dispatcher. *) Catch(0); ELSE Thread.Signal(d.changeState); END; WHILE d.state # d.desiredState DO Thread.Wait(d.mu, d.stateChanged); <== this never returns END; END; END; END ChangeState; It's a bit wierd to be mixing fork() and Modula-3 Thread? Or maybe it is ok? See, they are asking another process, from the fork() point of view, to change the state. It does so, but the write is private. Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? sbrk is maybe shared? mmap(anon|private) is not. Right now I have #ifndef apple sbrk #else mmap(anon|shared) #endif and it gets further. Hit an assertion failure in pthread. I'll try again. Cleanup all my RTIO. Maybe this notion of using fork() is just not supportable? In either case...you could paint it as an m3core problem, but ?it won't affect much code?. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: cvsup Date: Tue, 16 Mar 2010 12:32:49 +0000 I can reproduce the cvsup problem. An important point to consider is that cvsupd forks a server? Maybe that messes up initialization? I'll dig further. I think these steps are complete. I'll test them on a second clean system. You don't need any real data. jay at xlin2:~$ cat cvsupd.cfg *default tag=. *default host=xlin2. *default prefix=/home/jay *default base=/home/jay/var/db *default release=cvs delete use-rel-suffix compress src-all mkdir -p $HOME/var/db mkdir -p $HOME/data/cvsupd pkill cvsupd # cleanup any previous You don't really need any data. cvsup/cvsupd will issue an error in the "working" case, and hang in the non-working case. run the server: jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd 2010.03.16 05:24:25 PDT [22555]: CVSup server started 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 2010.03.16 05:24:25 PDT [22555]: Ready to service requests run the client: /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg Parsing supfile "/home/jay/cvsupd.cfg" Connecting to xlin2. Connected to xlin2. Server software version: 2010-03-05 Negotiating file attribute support Exchanging collection information Server message: Unknown collection "src-all" Establishing multiplexed-mode data connection Running Skipping collection src-all/cvs Shutting down connection to server Finished successfully it is able to talk to the server, and then fails. Ok -- at least it talked to the server. The server then exits too. I think that is correct (read the usage on the -C option). Then try with -C. The bug says -C 2. Let's use -C 1. They behave the same for our purposes. Just add -C 1 to the above server. Run the same client. The client hangs -- it never connects to the server. I reproduced this on LINUXLIBC6 and PPC_LINUX. Maybe more soon. I'm just rebuilding first. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 17:42:22 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 12:42:22 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: References: , , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , , , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com>, , , <20100316162842.qzx2mmd7cc8kockc@mail.elegosoft.com> Message-ID: <1C2120B5-5EC7-4850-8704-B37EFA882493@cs.purdue.edu> I think this would be a huge mess. Not worth pursuing. On 16 Mar 2010, at 11:51, Jay K wrote: > The "experiment" would also probably have to grow ADDRESS. > And then for interop I'd have to say, like HANDLE = BITS 32 FOR ADDRESS, if that is allowed. > And then..it gets confused..where would I get "32" from? > Compiler would have to truncate/extend pointers. Not sure it is willing to. > > Another good theoretical route is, indeed widen all the pointers, and then #ifdef the C code to take UINT64 and cast to pointers..and convert structs back/forth..but for Win32 we generally don't have such C code as we do for Posix. Darn. > I'll think about it more later but maybe it doesn't really work out. > > > - Jay > > > > Date: Tue, 16 Mar 2010 16:28:42 +0100 > > From: wagner at elegosoft.com > > To: hosking at cs.purdue.edu > > CC: jay.krell at cornell.edu; m3devel at elegosoft.com > > Subject: Re: [M3devel] release status [was something else] > > > > Quoting Tony Hosking : > > > > > Ah, yes, one issue about bringing over m3front changes is that it > > > also includes the atomics support. I don't think we want to do this > > > in this release. > > > So, this argues that we hold off on releasing the NT386 64-bit > > > LONGINT support for now. > > > > > > Thoughts? > > > > It's OK by me. You have convinced me that there are too many dependencies > > and implications. > > > > We should be able to start a new RC build in a few days then. > > > > Olaf > > -- > > Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 17:43:41 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 12:43:41 -0400 Subject: [M3devel] cvsup In-Reply-To: References: , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> Message-ID: <68E3ECCB-9F5E-4E4C-B8C2-FF1C14B32448@cs.purdue.edu> I thought I did a year or so back. But not certain. On 16 Mar 2010, at 12:10, Jay K wrote: > I could have sworn I had not seen hanging with sbrk or map_shared, but I can't get that now. > I'll dig a bit more..maybe not today. > To be clear, this isn't even the usual fork+exec. This is fork + do work + exit. > We can probably fix it by having it create a Modula-3 thread. > The first fork (daemonize) is probably fine. It is the per-client one that I think is a problem. > I'd like to find a theory as to why it ever worked, maybe see it work, and then fix it differently. > Maybe sbrk + user threads? > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > - Jay > > From: hosking at cs.purdue.edu > Date: Tue, 16 Mar 2010 11:43:52 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > mmap is strongly preferred over sbrk. > > I'd need to understand the control flow here in and across the processes, but yes, generally, mixing pthreads with fork may require significant care. > > On 16 Mar 2010, at 11:33, Jay K wrote: > > Daniel thank you for the bug report. > Thank you for suggesting strace. I used strace > and compared working vs. non-working. > And started added RTIO everywhere. > You can also use cvsup -f to slightly simplify -- one fork instead of two. > gdb set follow mode didn't seem to help. > > > I almost have it nailed down. > > > in CVSup, FSServer.m3, this code: > > FINALLY > IF isChild THEN > SigHandler.ShutDown(); <== > ELSE > SigHandler.Unblock(); > END; > END; > > > which runs fairly early, never returns in the child. > > > It ends up here: > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > (* Ask the dispatcher thread to change to a new state, and wait until > it has made the change. *) > BEGIN > LOCK d.mu DO > d.desiredState := state; > IF d.state # d.desiredState THEN > IF d.state = State.Running THEN > (* Send dummy signal 0 to wake up the dispatcher. *) > Catch(0); > ELSE > Thread.Signal(d.changeState); > END; > WHILE d.state # d.desiredState DO > Thread.Wait(d.mu, d.stateChanged); <== this never returns > END; > END; > END; > END ChangeState; > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > Or maybe it is ok? > > > See, they are asking another process, from the fork() point of view, to change the state. > It does so, but the write is private. > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > sbrk is maybe shared? mmap(anon|private) is not. > > Right now I have > #ifndef apple > sbrk > #else > mmap(anon|shared) > #endif > > > and it gets further. > Hit an assertion failure in pthread. > I'll try again. > Cleanup all my RTIO. > > > Maybe this notion of using fork() is just not supportable? > > > In either case...you could paint it as an m3core problem, but > ?it won't affect much code?. > > > - Jay > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Subject: cvsup > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > I can reproduce the cvsup problem. > An important point to consider is that cvsupd forks a server? > Maybe that messes up initialization? > I'll dig further. > > > I think these steps are complete. > I'll test them on a second clean system. > You don't need any real data. > > > jay at xlin2:~$ cat cvsupd.cfg > *default tag=. > *default host=xlin2. > *default prefix=/home/jay > *default base=/home/jay/var/db > *default release=cvs delete use-rel-suffix compress > > src-all > > > mkdir -p $HOME/var/db > mkdir -p $HOME/data/cvsupd > pkill cvsupd # cleanup any previous > > > You don't really need any data. > cvsup/cvsupd will issue an error in the "working" case, and > hang in the non-working case. > > > > > run the server: > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > run the client: > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > Parsing supfile "/home/jay/cvsupd.cfg" > Connecting to xlin2. > Connected to xlin2. > Server software version: 2010-03-05 > Negotiating file attribute support > Exchanging collection information > Server message: Unknown collection "src-all" > Establishing multiplexed-mode data connection > Running > Skipping collection src-all/cvs > Shutting down connection to server > Finished successfully > > it is able to talk to the server, and then fails. > Ok -- at least it talked to the server. > > The server then exits too. > I think that is correct (read the usage on the -C option). > > > Then try with -C. > The bug says -C 2. Let's use -C 1. > They behave the same for our purposes. > > > Just add -C 1 to the above server. > Run the same client. > The client hangs -- it never connects to the server. > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > Maybe more soon. > I'm just rebuilding first. > > > - Jay > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 16 18:07:38 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 18:07:38 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, Message-ID: <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> Quoting Jay K : > > User threads is apparently sufficient to let it work -- no change to > sbrk vs. mmap. > > You end up with a wierd mix of shared and not shared data, right? > > I'll try to make it use pthreads and Modula-3 threads and not fork(). I cannot really follow you here. Why should fork not work for threaded processes? The kernel should duplicate all threads and all memory regions, right? I don't seem to get the connection you make to sbrk vs. mmap. Can you explain? Using a thread for the process fork will probably not work contrary to my former statement that it might, as each client process is expected to have multiple threads. And the address spaces of the server processes should be separate for security and robustness reasons, as they are each connected to different clients, and one client's server crash must not impact the other sessions. Or am I misunderstanding what you intend to do? Olaf > - Jay > > > > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Tue, 16 Mar 2010 16:10:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > > > I could have sworn I had not seen hanging with sbrk or map_shared, > but I can't get that now. > I'll dig a bit more..maybe not today. > To be clear, this isn't even the usual fork+exec. This is fork + do > work + exit. > We can probably fix it by having it create a Modula-3 thread. > The first fork (daemonize) is probably fine. It is the per-client > one that I think is a problem. > I'd like to find a theory as to why it ever worked, maybe see it > work, and then fix it differently. > Maybe sbrk + user threads? > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > - Jay > > > > From: hosking at cs.purdue.edu > Date: Tue, 16 Mar 2010 11:43:52 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > mmap is strongly preferred over sbrk. > > > I'd need to understand the control flow here in and across the > processes, but yes, generally, mixing pthreads with fork may require > significant care. > > > > On 16 Mar 2010, at 11:33, Jay K wrote: > > Daniel thank you for the bug report. > Thank you for suggesting strace. I used strace > and compared working vs. non-working. > And started added RTIO everywhere. > You can also use cvsup -f to slightly simplify -- one fork instead of two. > gdb set follow mode didn't seem to help. > > > I almost have it nailed down. > > > in CVSup, FSServer.m3, this code: > > FINALLY > IF isChild THEN > SigHandler.ShutDown(); <== > ELSE > SigHandler.Unblock(); > END; > END; > > > which runs fairly early, never returns in the child. > > > It ends up here: > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > (* Ask the dispatcher thread to change to a new state, and wait until > it has made the change. *) > BEGIN > LOCK d.mu DO > d.desiredState := state; > IF d.state # d.desiredState THEN > IF d.state = State.Running THEN > (* Send dummy signal 0 to wake up the dispatcher. *) > Catch(0); > ELSE > Thread.Signal(d.changeState); > END; > WHILE d.state # d.desiredState DO > Thread.Wait(d.mu, d.stateChanged); <== this never returns > END; > END; > END; > END ChangeState; > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > Or maybe it is ok? > > > See, they are asking another process, from the fork() point of view, > to change the state. > It does so, but the write is private. > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > sbrk is maybe shared? mmap(anon|private) is not. > > Right now I have > #ifndef apple > sbrk > #else > mmap(anon|shared) > #endif > > > and it gets further. > Hit an assertion failure in pthread. > I'll try again. > Cleanup all my RTIO. > > > Maybe this notion of using fork() is just not supportable? > > > In either case...you could paint it as an m3core problem, but > ?it won't affect much code?. > > > - Jay > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Subject: cvsup > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > I can reproduce the cvsup problem. > An important point to consider is that cvsupd forks a server? > Maybe that messes up initialization? > I'll dig further. > > > I think these steps are complete. > I'll test them on a second clean system. > You don't need any real data. > > > jay at xlin2:~$ cat cvsupd.cfg > *default tag=. > *default host=xlin2. > *default prefix=/home/jay > *default base=/home/jay/var/db > *default release=cvs delete use-rel-suffix compress > > src-all > > > mkdir -p $HOME/var/db > mkdir -p $HOME/data/cvsupd > pkill cvsupd # cleanup any previous > > > You don't really need any data. > cvsup/cvsupd will issue an error in the "working" case, and > hang in the non-working case. > > > > > run the server: > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > run the client: > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > Parsing supfile "/home/jay/cvsupd.cfg" > Connecting to xlin2. > Connected to xlin2. > Server software version: 2010-03-05 > Negotiating file attribute support > Exchanging collection information > Server message: Unknown collection "src-all" > Establishing multiplexed-mode data connection > Running > Skipping collection src-all/cvs > Shutting down connection to server > Finished successfully > > it is able to talk to the server, and then fails. > Ok -- at least it talked to the server. > > The server then exits too. > I think that is correct (read the usage on the -C option). > > > Then try with -C. > The bug says -C 2. Let's use -C 1. > They behave the same for our purposes. > > > Just add -C 1 to the above server. > Run the same client. > The client hangs -- it never connects to the server. > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > Maybe more soon. > I'm just rebuilding first. > > > - Jay > > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 16 18:43:40 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 17:43:40 +0000 Subject: [M3devel] cvsup In-Reply-To: <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> Message-ID: With fork, which data is copy-on-write vs. shared-write? mmap has a flag, so it is up to each caller. You end up with a mix. Whether or not you get one forked thread or all of them I believe varies. Solaris has "fork1()". There is no obvious answer. It just goes to show how wierd fork() is. I'm not sure any longer there is a connection to mmap/sbrk. It might be just user threads vs. not. Though there again there is a question as to which data is shared-write vs. copy-on-write. The server should not crash. It is written in an optionally save language, after all. More so, that is I believe a much more larger more difficult fix to apply. Using processes for such separation is deemed overkill, depending on whose story you believe. I'll have to think about it more. - Jay > Date: Tue, 16 Mar 2010 18:07:38 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > Quoting Jay K : > > > > > User threads is apparently sufficient to let it work -- no change to > > sbrk vs. mmap. > > > > You end up with a wierd mix of shared and not shared data, right? > > > > I'll try to make it use pthreads and Modula-3 threads and not fork(). > > I cannot really follow you here. Why should fork not work for > threaded processes? The kernel should duplicate all threads and all > memory regions, right? > > I don't seem to get the connection you make to sbrk vs. mmap. > Can you explain? > > Using a thread for the process fork will probably not work contrary > to my former statement that it might, as each client process is > expected to have multiple threads. > > And the address spaces of the server processes should be separate > for security and robustness reasons, as they are each connected to > different clients, and one client's server crash must not impact the > other sessions. > > Or am I misunderstanding what you intend to do? > > Olaf > > > - Jay > > > > > > > > > > From: jay.krell at cornell.edu > > To: hosking at cs.purdue.edu > > Date: Tue, 16 Mar 2010 16:10:15 +0000 > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] cvsup > > > > > > > > I could have sworn I had not seen hanging with sbrk or map_shared, > > but I can't get that now. > > I'll dig a bit more..maybe not today. > > To be clear, this isn't even the usual fork+exec. This is fork + do > > work + exit. > > We can probably fix it by having it create a Modula-3 thread. > > The first fork (daemonize) is probably fine. It is the per-client > > one that I think is a problem. > > I'd like to find a theory as to why it ever worked, maybe see it > > work, and then fix it differently. > > Maybe sbrk + user threads? > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > > > - Jay > > > > > > > > From: hosking at cs.purdue.edu > > Date: Tue, 16 Mar 2010 11:43:52 -0400 > > To: jay.krell at cornell.edu > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] cvsup > > > > mmap is strongly preferred over sbrk. > > > > > > I'd need to understand the control flow here in and across the > > processes, but yes, generally, mixing pthreads with fork may require > > significant care. > > > > > > > > On 16 Mar 2010, at 11:33, Jay K wrote: > > > > Daniel thank you for the bug report. > > Thank you for suggesting strace. I used strace > > and compared working vs. non-working. > > And started added RTIO everywhere. > > You can also use cvsup -f to slightly simplify -- one fork instead of two. > > gdb set follow mode didn't seem to help. > > > > > > I almost have it nailed down. > > > > > > in CVSup, FSServer.m3, this code: > > > > FINALLY > > IF isChild THEN > > SigHandler.ShutDown(); <== > > ELSE > > SigHandler.Unblock(); > > END; > > END; > > > > > > which runs fairly early, never returns in the child. > > > > > > It ends up here: > > > > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > > (* Ask the dispatcher thread to change to a new state, and wait until > > it has made the change. *) > > BEGIN > > LOCK d.mu DO > > d.desiredState := state; > > IF d.state # d.desiredState THEN > > IF d.state = State.Running THEN > > (* Send dummy signal 0 to wake up the dispatcher. *) > > Catch(0); > > ELSE > > Thread.Signal(d.changeState); > > END; > > WHILE d.state # d.desiredState DO > > Thread.Wait(d.mu, d.stateChanged); <== this never returns > > END; > > END; > > END; > > END ChangeState; > > > > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > > Or maybe it is ok? > > > > > > See, they are asking another process, from the fork() point of view, > > to change the state. > > It does so, but the write is private. > > > > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > > sbrk is maybe shared? mmap(anon|private) is not. > > > > Right now I have > > #ifndef apple > > sbrk > > #else > > mmap(anon|shared) > > #endif > > > > > > and it gets further. > > Hit an assertion failure in pthread. > > I'll try again. > > Cleanup all my RTIO. > > > > > > Maybe this notion of using fork() is just not supportable? > > > > > > In either case...you could paint it as an m3core problem, but > > ?it won't affect much code?. > > > > > > - Jay > > > > > > > > From: jay.krell at cornell.edu > > To: m3devel at elegosoft.com > > Subject: cvsup > > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > > > I can reproduce the cvsup problem. > > An important point to consider is that cvsupd forks a server? > > Maybe that messes up initialization? > > I'll dig further. > > > > > > I think these steps are complete. > > I'll test them on a second clean system. > > You don't need any real data. > > > > > > jay at xlin2:~$ cat cvsupd.cfg > > *default tag=. > > *default host=xlin2. > > *default prefix=/home/jay > > *default base=/home/jay/var/db > > *default release=cvs delete use-rel-suffix compress > > > > src-all > > > > > > mkdir -p $HOME/var/db > > mkdir -p $HOME/data/cvsupd > > pkill cvsupd # cleanup any previous > > > > > > You don't really need any data. > > cvsup/cvsupd will issue an error in the "working" case, and > > hang in the non-working case. > > > > > > > > > > run the server: > > > > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > > > > run the client: > > > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > > > > Parsing supfile "/home/jay/cvsupd.cfg" > > Connecting to xlin2. > > Connected to xlin2. > > Server software version: 2010-03-05 > > Negotiating file attribute support > > Exchanging collection information > > Server message: Unknown collection "src-all" > > Establishing multiplexed-mode data connection > > Running > > Skipping collection src-all/cvs > > Shutting down connection to server > > Finished successfully > > > > it is able to talk to the server, and then fails. > > Ok -- at least it talked to the server. > > > > The server then exits too. > > I think that is correct (read the usage on the -C option). > > > > > > Then try with -C. > > The bug says -C 2. Let's use -C 1. > > They behave the same for our purposes. > > > > > > Just add -C 1 to the above server. > > Run the same client. > > The client hangs -- it never connects to the server. > > > > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > > Maybe more soon. > > I'm just rebuilding first. > > > > > > - Jay > > > > > > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 20:32:52 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 15:32:52 -0400 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> Message-ID: fork should still work with pthreads. Why wouldn't it? Unless there is something weird about the semaphore used to start and stop threads? If the child ends up with the same semaphore then that would be a problem. Perhaps we need to ensure it is replicated it in forked children using pthread_atfork? You could try things on a target like OSX where threads are stopped/started natively (i.e., no semaphore being used). Though, on inspection I see that sem_init is called with pshared == false. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 16 Mar 2010, at 13:43, Jay K wrote: > With fork, which data is copy-on-write vs. shared-write? > mmap has a flag, so it is up to each caller. You end up with a mix. > > > Whether or not you get one forked thread or all of them I believe varies. > Solaris has "fork1()". > There is no obvious answer. > It just goes to show how wierd fork() is. > > > I'm not sure any longer there is a connection to mmap/sbrk. > It might be just user threads vs. not. > Though there again there is a question as to which data is > shared-write vs. copy-on-write. > > > The server should not crash. > It is written in an optionally save language, after all. > More so, that is I believe a much more larger more difficult fix to apply. > Using processes for such separation is deemed overkill, depending > on whose story you believe. > I'll have to think about it more. > > > - Jay > > > Date: Tue, 16 Mar 2010 18:07:38 +0100 > > From: wagner at elegosoft.com > > To: m3devel at elegosoft.com > > Subject: Re: [M3devel] cvsup > > > > Quoting Jay K : > > > > > > > > User threads is apparently sufficient to let it work -- no change to > > > sbrk vs. mmap. > > > > > > You end up with a wierd mix of shared and not shared data, right? > > > > > > I'll try to make it use pthreads and Modula-3 threads and not fork(). > > > > I cannot really follow you here. Why should fork not work for > > threaded processes? The kernel should duplicate all threads and all > > memory regions, right? > > > > I don't seem to get the connection you make to sbrk vs. mmap. > > Can you explain? > > > > Using a thread for the process fork will probably not work contrary > > to my former statement that it might, as each client process is > > expected to have multiple threads. > > > > And the address spaces of the server processes should be separate > > for security and robustness reasons, as they are each connected to > > different clients, and one client's server crash must not impact the > > other sessions. > > > > Or am I misunderstanding what you intend to do? > > > > Olaf > > > > > - Jay > > > > > > > > > > > > > > > From: jay.krell at cornell.edu > > > To: hosking at cs.purdue.edu > > > Date: Tue, 16 Mar 2010 16:10:15 +0000 > > > CC: m3devel at elegosoft.com > > > Subject: Re: [M3devel] cvsup > > > > > > > > > > > > I could have sworn I had not seen hanging with sbrk or map_shared, > > > but I can't get that now. > > > I'll dig a bit more..maybe not today. > > > To be clear, this isn't even the usual fork+exec. This is fork + do > > > work + exit. > > > We can probably fix it by having it create a Modula-3 thread. > > > The first fork (daemonize) is probably fine. It is the per-client > > > one that I think is a problem. > > > I'd like to find a theory as to why it ever worked, maybe see it > > > work, and then fix it differently. > > > Maybe sbrk + user threads? > > > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > > > > > - Jay > > > > > > > > > > > > From: hosking at cs.purdue.edu > > > Date: Tue, 16 Mar 2010 11:43:52 -0400 > > > To: jay.krell at cornell.edu > > > CC: m3devel at elegosoft.com > > > Subject: Re: [M3devel] cvsup > > > > > > mmap is strongly preferred over sbrk. > > > > > > > > > I'd need to understand the control flow here in and across the > > > processes, but yes, generally, mixing pthreads with fork may require > > > significant care. > > > > > > > > > > > > On 16 Mar 2010, at 11:33, Jay K wrote: > > > > > > Daniel thank you for the bug report. > > > Thank you for suggesting strace. I used strace > > > and compared working vs. non-working. > > > And started added RTIO everywhere. > > > You can also use cvsup -f to slightly simplify -- one fork instead of two. > > > gdb set follow mode didn't seem to help. > > > > > > > > > I almost have it nailed down. > > > > > > > > > in CVSup, FSServer.m3, this code: > > > > > > FINALLY > > > IF isChild THEN > > > SigHandler.ShutDown(); <== > > > ELSE > > > SigHandler.Unblock(); > > > END; > > > END; > > > > > > > > > which runs fairly early, never returns in the child. > > > > > > > > > It ends up here: > > > > > > > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > > > (* Ask the dispatcher thread to change to a new state, and wait until > > > it has made the change. *) > > > BEGIN > > > LOCK d.mu DO > > > d.desiredState := state; > > > IF d.state # d.desiredState THEN > > > IF d.state = State.Running THEN > > > (* Send dummy signal 0 to wake up the dispatcher. *) > > > Catch(0); > > > ELSE > > > Thread.Signal(d.changeState); > > > END; > > > WHILE d.state # d.desiredState DO > > > Thread.Wait(d.mu, d.stateChanged); <== this never returns > > > END; > > > END; > > > END; > > > END ChangeState; > > > > > > > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > > > Or maybe it is ok? > > > > > > > > > See, they are asking another process, from the fork() point of view, > > > to change the state. > > > It does so, but the write is private. > > > > > > > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > > > sbrk is maybe shared? mmap(anon|private) is not. > > > > > > Right now I have > > > #ifndef apple > > > sbrk > > > #else > > > mmap(anon|shared) > > > #endif > > > > > > > > > and it gets further. > > > Hit an assertion failure in pthread. > > > I'll try again. > > > Cleanup all my RTIO. > > > > > > > > > Maybe this notion of using fork() is just not supportable? > > > > > > > > > In either case...you could paint it as an m3core problem, but > > > ?it won't affect much code?. > > > > > > > > > - Jay > > > > > > > > > > > > From: jay.krell at cornell.edu > > > To: m3devel at elegosoft.com > > > Subject: cvsup > > > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > > > > > I can reproduce the cvsup problem. > > > An important point to consider is that cvsupd forks a server? > > > Maybe that messes up initialization? > > > I'll dig further. > > > > > > > > > I think these steps are complete. > > > I'll test them on a second clean system. > > > You don't need any real data. > > > > > > > > > jay at xlin2:~$ cat cvsupd.cfg > > > *default tag=. > > > *default host=xlin2. > > > *default prefix=/home/jay > > > *default base=/home/jay/var/db > > > *default release=cvs delete use-rel-suffix compress > > > > > > src-all > > > > > > > > > mkdir -p $HOME/var/db > > > mkdir -p $HOME/data/cvsupd > > > pkill cvsupd # cleanup any previous > > > > > > > > > You don't really need any data. > > > cvsup/cvsupd will issue an error in the "working" case, and > > > hang in the non-working case. > > > > > > > > > > > > > > > run the server: > > > > > > > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > > > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > > > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > > > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > > > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > > > > > > > run the client: > > > > > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > > > > > > > Parsing supfile "/home/jay/cvsupd.cfg" > > > Connecting to xlin2. > > > Connected to xlin2. > > > Server software version: 2010-03-05 > > > Negotiating file attribute support > > > Exchanging collection information > > > Server message: Unknown collection "src-all" > > > Establishing multiplexed-mode data connection > > > Running > > > Skipping collection src-all/cvs > > > Shutting down connection to server > > > Finished successfully > > > > > > it is able to talk to the server, and then fails. > > > Ok -- at least it talked to the server. > > > > > > The server then exits too. > > > I think that is correct (read the usage on the -C option). > > > > > > > > > Then try with -C. > > > The bug says -C 2. Let's use -C 1. > > > They behave the same for our purposes. > > > > > > > > > Just add -C 1 to the above server. > > > Run the same client. > > > The client hangs -- it never connects to the server. > > > > > > > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > > > Maybe more soon. > > > I'm just rebuilding first. > > > > > > > > > - Jay > > > > > > > > > > > > > > -- > > Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 16 21:59:12 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 21:59:12 +0100 Subject: [M3devel] make-deb.py Message-ID: <20100316215912.bvvkuei9wk84w8ks@mail.elegosoft.com> I was wondering why there are no Debian or .msi packages, and now have this trace: + type python python is /usr/local/bin/python + [ xAMD64_FREEBSD = xNT386 ] + python /ad0e/home/wagner/work/cm3/scripts/python/make-deb.py /var/tmp/cm3 set CM3_TARGET=AMD64_FREEBSD set CM3_INSTALL=/var/tmp/cm3 set M3CONFIG=/ad0e/home/wagner/work/cm3/m3-sys/cminstall/src/config-no-install/AMD64_FREEBSD Traceback (most recent call last): File "/ad0e/home/wagner/work/cm3/scripts/python/make-deb.py", line 12, in MakeDebianPackage(sys.argv[1], "/usr/local/cm3") TypeError: MakeDebianPackage() takes exactly 4 arguments (2 given) + mv /var/tmp/cm3.deb /var/tmp/cm3-AMD64_FREEBSD-pre-RC5.deb mv: rename /var/tmp/cm3.deb to /var/tmp/cm3-AMD64_FREEBSD-pre-RC5.deb: No such f It seems the scripts are broken or inconsistent in the release branch. Can you please have a look? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From m3 at sol42.com Tue Mar 16 22:19:26 2010 From: m3 at sol42.com (Daniel Solaz) Date: Tue, 16 Mar 2010 22:19:26 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> Message-ID: On 16 Mar 2010, at 20:32, Tony Hosking wrote: > fork should still work with pthreads. Why wouldn't it? Chapter 6.1 of Butenhof's pthreads book deals with this, and well, it is a mess. According to him only the calling thread exists on return from fork() in the child process, but the other pthread *states* (mutexes, conditions, data keys, etc.) are replicated as well. And I'm sure the details vary from system to system. This is the chapter's first sentence: "Avoid using fork in a threaded program (if you can) unless you intend to exec a new program immediately." Regards. -Daniel From rcolebur at SCIRES.COM Wed Mar 17 01:27:39 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Tue, 16 Mar 2010 20:27:39 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: I'm still catching up on my m3devel reading. I was hoping we would put the 64-bit stuff in for NT386. But, I don't want to hold up the release if this is going to be problematic. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Tuesday, March 16, 2010 11:22 AM To: Jay K Cc: m3devel Subject: Re: [M3devel] release status [was something else] Ah, yes, one issue about bringing over m3front changes is that it also includes the atomics support. I don't think we want to do this in this release. So, this argues that we hold off on releasing the NT386 64-bit LONGINT support for now. Thoughts? ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Wed Mar 17 11:35:32 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Wed, 17 Mar 2010 06:35:32 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: References: <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: <20100317103531.GB18479@topoi.pooq.com> On Tue, Mar 16, 2010 at 08:27:39PM -0400, Coleburn, Randy wrote: > I'm still catching up on my m3devel reading. > I was hoping we would put the 64-bit stuff in for NT386. > But, I don't want to hold up the release if this is going to be problematic. > Regards, > Randy Judging from the discussions here, the biggest problem in building a release was that it hadn't been done for some time, and the mechanisms for doing it were all rusty. Well, it shouldn't take as long for hte next one. It's mostly well-oiled by now. -- hendrik From jay.krell at cornell.edu Wed Mar 17 08:47:45 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 07:47:45 +0000 Subject: [M3devel] fork/cvsup Message-ID: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Wed Mar 17 10:19:59 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Wed, 17 Mar 2010 10:19:59 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> Message-ID: <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com> Quoting Daniel Solaz : > On 16 Mar 2010, at 20:32, Tony Hosking wrote: >> fork should still work with pthreads. Why wouldn't it? > Chapter 6.1 of Butenhof's pthreads book deals with this, and well, > it is a mess. According to him only the calling thread exists on > return from fork() in the child process, but the other pthread > *states* (mutexes, conditions, data keys, etc.) are replicated as > well. And I'm sure the details vary from system to system. > This is the chapter's first sentence: "Avoid using fork in a > threaded program (if you can) unless you intend to exec a new > program immediately." Is this about kernel threads or user level threads? Or about a specific implementation of one or the other? I don't know the book, but I always thought about pthreads as being the POSIX specification of threads, and I don't think there's anything in that relating to fork, AFAIR, but that was years ago. Can anyone enlighten us about the POSIX specs and if they define the semantics of fork in presence of threaded processes? Or does it really boil down to the above `avoid it if you can'? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Wed Mar 17 11:07:00 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Wed, 17 Mar 2010 11:07:00 +0100 Subject: [M3devel] Open issues (trac tickets) for cm3 release 5.8 Message-ID: <20100317110700.z1b1sipi80oks8kc@mail.elegosoft.com> There are still a number of open tickets regarding the release in our trac system. Please have a look at https://projects.elego.de/cm3/query?status=resolved&status=reopened&status=assigned&status=analyzed&status=new&status=accepted&group=status&milestone=CM3+release+5.8 I think several of these have already been addressed and can be closed, and others can and should possibly be postponed. Tickets that are assigned to me usually just haven't been assigned to anybody else yet, as I'm the default for that field, so don't ignore them. I can manage these entries, help with information, give you access rights, but haven't the time to do the actual work except in one or two cases. So please try to solve and close as many of the issues as you can before the release. I may also add that analyzing and fixing bugs is always a good starting point for newcomers and those not involved actively so far and just lurking for something to do :-) If you have any questions, don't hesitate to ask me, Olaf PS: Some time ago, there were several requests for trac to coordinate our development. Until now it hasn't really extensively been used; but we should change that. It can really help us to coordinate our work and document changes and their history. It's WiKi can also be used to add various information that is not as formal as tickets and milestones. -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Wed Mar 17 11:42:06 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 10:42:06 +0000 Subject: [M3devel] cvsup In-Reply-To: <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com> References: , , ,,<2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, , , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com> Message-ID: Please start here: http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html "There are at least two serious problems with the semantics of fork() in a multi-threaded program" I've been looking at cvsup a while now. Lots of RTIO.PutText, etc. pthread_atfork usage in RTCollector and ThreadPThread.m3. Reinitialize in the child, mark the gc threads as not being created yet. Been using @M3nogc. It helps. I think I understand why. I have a somewhat wild educated guess as to the problem. It is "fork1" vs. "forkall", sort of. In user threads, when you fork(), you get all the user Modula-3 threads continuing to run in the child. Because their existance is an artifact of a bunch of data that we maintain and gets carried into the new process. With kernel threads (with the exception of using Solaris forkall()), you get just the thread that called fork(). See my reference to the RTCollector/Allocator atfork change, which I show here (not yet commited): PROCEDURE AtForkPrepare() = BEGIN END AtForkPrepare; PROCEDURE AtForkParent() = BEGIN END AtForkParent; PROCEDURE AtForkChild() = VAR r: INTEGER; BEGIN r := Thread.PThreadAtFork(AtForkPrepare, AtForkParent, AtForkChild); <* ASSERT r = 0 *> startedBackground := FALSE; startedForeground := FALSE; END AtForkChild; PROCEDURE Init () = VAR r: INTEGER; BEGIN r := Thread.PThreadAtFork(AtForkPrepare, AtForkParent, AtForkChild); <* ASSERT r = 0 *> Some threads may need to be recreated in the child, some not. The default is not. I think we mainly have to adjust cvsup to recreate its threads. And a little bit of pthread_atfork use in m3core (a bunch of it shown believe, and reinitialization in ThreadPThread.m3). Like the Apple/Darwin warnings, I don't think most libraries can be considered "fork safe". That is you can fork, but only if you exec soon thereafter. Making code "fork safe" I believe equates to reviewing it a bunch and using pthread_atfork selectively. One thing to watch for is if the library creates any worker threads during "initialization" or even "on demand" -- to either recreate them in the child, or note that they don't exist in the child. We can make m3core fork safe, I guess, to satisfy cvsup. Maybe write the code to duplicate all threads in a child, subject to some configuration setting? Something like @M3forkall, but it'd be something that e.g. cvsup would specify when it builds, and then user wouldn't have to list it anywhere. Kind of like the flags for incremental and vm gc that get recorded by the compiler. Anyway, let me see about recreating cvsup's threads manually. Probably by using the Thread.PThreadAtFork I added. I think the initial hang I described fits my theory -- the dispatcher thread doesn't exist in the child, so hang. And then when I avoid that with a local edit I get a hang later. I compared without -C and I see other stuff happening..I think on other threads that fork() is abandoning.. Thread.PThreadAtFork is wierd, but I think reasonable. It will do nothing on user threads and Win32. It will only do anything on PThreads. Libraries can use it to achieve compatibility with pthreads programs that use fork. - Jay > Date: Wed, 17 Mar 2010 10:19:59 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > Quoting Daniel Solaz : > > > On 16 Mar 2010, at 20:32, Tony Hosking wrote: > >> fork should still work with pthreads. Why wouldn't it? > > Chapter 6.1 of Butenhof's pthreads book deals with this, and well, > > it is a mess. According to him only the calling thread exists on > > return from fork() in the child process, but the other pthread > > *states* (mutexes, conditions, data keys, etc.) are replicated as > > well. And I'm sure the details vary from system to system. > > This is the chapter's first sentence: "Avoid using fork in a > > threaded program (if you can) unless you intend to exec a new > > program immediately." > > Is this about kernel threads or user level threads? Or about a specific > implementation of one or the other? > > I don't know the book, but I always thought about pthreads as being the > POSIX specification of threads, and I don't think there's anything > in that relating to fork, AFAIR, but that was years ago. > > Can anyone enlighten us about the POSIX specs and if they define the > semantics of fork in presence of threaded processes? Or does it really > boil down to the above `avoid it if you can'? > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 11:50:31 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 10:50:31 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , ,,,,<2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>,,, , , ,,, , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, , , , , , , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com>, Message-ID: Here's a shorter more direct version. Please start here: http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html "There are at least two serious problems with the semantics of fork() in a multi-threaded program" I have a good theory as to the problem and the fix. Problem: With user threads, when you fork(), all the threads keep running in the new child. Because the data that causes them to exist is carried forward. With kernel threads, they don't, just the caller of fork(). Fix: A few small uses of pthread_atfork both in m3core and cvsup. Abstracted behind Thread.PThreadAtFork that does nothing for user threads and Win32. They would do things like "reinitialize globals" and "recreate worker threads". You can't just call all the module initializers over. That would ultimately I believe reset too much data. ? It might be possible, though, to setjmp, run the module initializers, longjmp. Something like, perhaps, a flag to RTLinker.InitRuntime that causes it to longjmp instead of calling main. However this would still e.g. give the initial thread the wrong stackbase and mess up garbage collection. I'm much more keen on selective use of pthread_atfork, in m3core and cvsup. A more conventional approach is the program to rerun itself with some flags that "push" it fast to "resume" point, but that somewhat defeats the purpose of the fork + do work model -- cheap reuse of already established state. Lingering problem: Any libraries that create worker threads need to use Thread.PThreadAtFork in order to be compatible with the fork + do more work pattern. fork + exec is fine. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 12:02:02 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 11:02:02 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , , , , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , , , , , , , , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, ,,, ,,, ,,, , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com>, , , Message-ID: Another good option might be "RecreateThreadsAfterFork()" that cvsup can call. I'm trying that. If that doesn't work, I'll track down all of cvsup's thread creates. - Jay From: jay.krell at cornell.edu To: wagner at elegosoft.com; m3devel at elegosoft.com Date: Wed, 17 Mar 2010 10:50:31 +0000 Subject: Re: [M3devel] cvsup Here's a shorter more direct version. Please start here: http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html "There are at least two serious problems with the semantics of fork() in a multi-threaded program" I have a good theory as to the problem and the fix. Problem: With user threads, when you fork(), all the threads keep running in the new child. Because the data that causes them to exist is carried forward. With kernel threads, they don't, just the caller of fork(). Fix: A few small uses of pthread_atfork both in m3core and cvsup. Abstracted behind Thread.PThreadAtFork that does nothing for user threads and Win32. They would do things like "reinitialize globals" and "recreate worker threads". You can't just call all the module initializers over. That would ultimately I believe reset too much data. ? It might be possible, though, to setjmp, run the module initializers, longjmp. Something like, perhaps, a flag to RTLinker.InitRuntime that causes it to longjmp instead of calling main. However this would still e.g. give the initial thread the wrong stackbase and mess up garbage collection. I'm much more keen on selective use of pthread_atfork, in m3core and cvsup. A more conventional approach is the program to rerun itself with some flags that "push" it fast to "resume" point, but that somewhat defeats the purpose of the fork + do work model -- cheap reuse of already established state. Lingering problem: Any libraries that create worker threads need to use Thread.PThreadAtFork in order to be compatible with the fork + do more work pattern. fork + exec is fine. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 12:10:43 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 11:10:43 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , , , , , ,,<2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , ,,, , , , , ,,, , ,,, <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, , , , , , , , , , , , , , , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com>, , , , , , Message-ID: For that matter, we can just provide Thread.ForkAll. special implementation on pthreads. Just call Unix.fork() on user threads. No implementation for Win32. - Jay From: jay.krell at cornell.edu To: wagner at elegosoft.com; m3devel at elegosoft.com Date: Wed, 17 Mar 2010 11:02:02 +0000 Subject: Re: [M3devel] cvsup Another good option might be "RecreateThreadsAfterFork()" that cvsup can call. I'm trying that. If that doesn't work, I'll track down all of cvsup's thread creates. - Jay From: jay.krell at cornell.edu To: wagner at elegosoft.com; m3devel at elegosoft.com Date: Wed, 17 Mar 2010 10:50:31 +0000 Subject: Re: [M3devel] cvsup Here's a shorter more direct version. Please start here: http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html "There are at least two serious problems with the semantics of fork() in a multi-threaded program" I have a good theory as to the problem and the fix. Problem: With user threads, when you fork(), all the threads keep running in the new child. Because the data that causes them to exist is carried forward. With kernel threads, they don't, just the caller of fork(). Fix: A few small uses of pthread_atfork both in m3core and cvsup. Abstracted behind Thread.PThreadAtFork that does nothing for user threads and Win32. They would do things like "reinitialize globals" and "recreate worker threads". You can't just call all the module initializers over. That would ultimately I believe reset too much data. ? It might be possible, though, to setjmp, run the module initializers, longjmp. Something like, perhaps, a flag to RTLinker.InitRuntime that causes it to longjmp instead of calling main. However this would still e.g. give the initial thread the wrong stackbase and mess up garbage collection. I'm much more keen on selective use of pthread_atfork, in m3core and cvsup. A more conventional approach is the program to rerun itself with some flags that "push" it fast to "resume" point, but that somewhat defeats the purpose of the fork + do work model -- cheap reuse of already established state. Lingering problem: Any libraries that create worker threads need to use Thread.PThreadAtFork in order to be compatible with the fork + do more work pattern. fork + exec is fine. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Wed Mar 17 12:28:50 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Wed, 17 Mar 2010 12:28:50 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , ,,,,<2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>,,, , , ,,, , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, , , , , , , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com>, Message-ID: <20100317122850.gd98fzkfs0kokgs4@mail.elegosoft.com> Quoting Jay K : > Here's a shorter more direct version. > > Please start here: > http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html > "There are at least two serious problems with the semantics of > fork() in a multi-threaded program" > > I have a good theory as to the problem and the fix. > > Problem: > > With user threads, when you fork(), all the threads keep running in > the new child. > > Because the data that causes them to exist is carried forward. > > With kernel threads, they don't, just the caller of fork(). This is where I seem to have been wrong. I always thought that fork() would indeed duplicate all threads and their execution states. > Fix: > > A few small uses of pthread_atfork both in m3core and cvsup. > > Abstracted behind Thread.PThreadAtFork that does nothing for user > threads and Win32. > > They would do things like "reinitialize globals" and "recreate > worker threads". This sounds reasonable, but is of course not trivial. I think we should at least guarantee that the M3 runtime threads that were running are available again and will do their work. I'm not sure if cvsupd itself will need application extensions, or if the threads handling the streaming protocol are created after the fork. John Polstra used to be on this list and may be able to help. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Wed Mar 17 13:41:23 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 12:41:23 +0000 Subject: [M3devel] cvsup In-Reply-To: <20100317122850.gd98fzkfs0kokgs4@mail.elegosoft.com> References: , , , , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , , , , , , , , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, , , , , , , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com>, , , <20100317122850.gd98fzkfs0kokgs4@mail.elegosoft.com> Message-ID: Olaf, also search the web for fork1, forkall. Solaris has them. You were right for Modula-3 user threads. But not for Win32 or pthreads. I tried something by accident..and it seemed to work. In my attempts to write PThread.ForkAll, which would fork() and then in the child initialize and recreate the threads of the parent (except for the current thread), I commented out the actual creating, leaving in the initialization..it oddly, it worked. I probably had the "ShutDown" hack in cvsup (direct call instead of through the cvsup dispatcher). Perhaps this works because my cvsup test case is too small and doesn't require its other threads. Or maybe it suffices. Still, there are two approaches to a fix. They are both not difficult. One is to provide the overarching function: Thread.ForkAll(). Cvsup would just call that instead of Unix.fork. The other is to provide pthread_atfork wrapper, use it internally a little bit, and have cvsup use it as well to recreate its threads. But I'm not sure it needs to. Modula-3 wouldn't actually have to recreate any threads, just set the booleans in RTCollector to false indicating they haven't been created. Could be that cvsup doesn't require any/many threads to flow from server to forked server, maybe just the one that it tears down right away. - Jay > Date: Wed, 17 Mar 2010 12:28:50 +0100 > From: wagner at elegosoft.com > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] cvsup > > Quoting Jay K : > > > Here's a shorter more direct version. > > > > Please start here: > > http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html > > "There are at least two serious problems with the semantics of > > fork() in a multi-threaded program" > > > > I have a good theory as to the problem and the fix. > > > > Problem: > > > > With user threads, when you fork(), all the threads keep running in > > the new child. > > > > Because the data that causes them to exist is carried forward. > > > > With kernel threads, they don't, just the caller of fork(). > > This is where I seem to have been wrong. I always thought that fork() > would indeed duplicate all threads and their execution states. > > > Fix: > > > > A few small uses of pthread_atfork both in m3core and cvsup. > > > > Abstracted behind Thread.PThreadAtFork that does nothing for user > > threads and Win32. > > > > They would do things like "reinitialize globals" and "recreate > > worker threads". > > This sounds reasonable, but is of course not trivial. > I think we should at least guarantee that the M3 runtime threads > that were running are available again and will do their work. > > I'm not sure if cvsupd itself will need application extensions, or if > the threads handling the streaming protocol are created after the fork. > > John Polstra used to be on this list and may be able to help. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 17 14:28:14 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 17 Mar 2010 09:28:14 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: References: Message-ID: Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such time as one of the exec functions is > called. All APIs, including global data symbols, in any framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 15:01:15 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 14:01:15 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: References: , Message-ID: Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 16:39:59 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 15:39:59 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , Message-ID: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 17 19:01:31 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 17 Mar 2010 14:01:31 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , Message-ID: <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu> Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: > I have something working on Solaris now. > More details after testing on Linux and Darwin. > > - Jay > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Exec what? > You'd have to change the code to carefully reach the same place. > > - Jay > > Subject: Re: [M3devel] fork/cvsup > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 09:28:14 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not just exec in the child? > > On 17 Mar 2010, at 03:47, Jay K wrote: > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such time as one of the exec functions is > called. All APIs, including global data symbols, in any framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 19:13:45 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 18:13:45 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu> References: , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu> Message-ID: --- bad news: It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. Restart manually. Works again. Around 9 times. Then hangs again. That is on Linux/x86 and Solaris/sparc. Doesn't work at all on Mac/amd64, just hangs. --- sketch: m3core uses pthread_atfork to selectively reinitialize Mainly to only have one thread. common Thread.PThreadAtFork is provided for others to do the same It is deliberately in a portable interface. Thread.ReforkThreadAfterProcessFork Is provided for users to restart threads from their child AtFork hander. This is used by the allocator/collector. Thread.ForkProcessAndAllThreads() Is used by "lazy" clients who want to restart all their threads but didn't keep track of them. The runtime can do it for them. This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads or not, depending on if they need their threads restarted. The runtime takes care of its threads either way. --- What'd I'd written up: attached works typically 9 times on Linux and Solaris before server hangs again. No improvement on Darwin, just hangs. Can't see much in debuggers for some reason. There is extra allowance in the m3core change such that users of fork + do work (as opposed to fork + exec) may or may not call ForkAll, depending on if they feel a need for their own threads to be recreated, and if they've kept track of how to recreate them, or just rely on the runtime to know all the threads. There are three runtime threads that are sometimes created in the parent, and if so, recreated in the child. background collector, foreground collector, weak ref thread I'll try to poke at it some more. I'm not sure what is the best way to suspend all threads. I tried a few differnt ways. SuspendOthers LockHeap pthread_mutex_lock various combinations It is deliberate that pthread specific code is in common/Thread.i3. That way code can be portable, at least among the two Posix thread implementations. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:31 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: m3core_atfork.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cvsup_forkall.txt URL: From hosking at cs.purdue.edu Wed Mar 17 19:30:47 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 17 Mar 2010 14:30:47 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu> Message-ID: <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu> I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 14:13, Jay K wrote: > --- > bad news: > It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. > Restart manually. Works again. Around 9 times. Then hangs again. > That is on Linux/x86 and Solaris/sparc. > Doesn't work at all on Mac/amd64, just hangs. > > --- > sketch: > m3core uses pthread_atfork to selectively reinitialize > Mainly to only have one thread. > > > common Thread.PThreadAtFork is provided for others to do the same > It is deliberately in a portable interface. > > > Thread.ReforkThreadAfterProcessFork > Is provided for users to restart threads from their child AtFork hander. > This is used by the allocator/collector. > > > Thread.ForkProcessAndAllThreads() > Is used by "lazy" clients who want to restart all their threads > but didn't keep track of them. The runtime can do it for them. > > > This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads > or not, depending on if they need their threads restarted. > The runtime takes care of its threads either way. > > > --- > What'd I'd written up: > > attached works typically 9 times on Linux and Solaris > before server hangs again. > > > No improvement on Darwin, just hangs. > Can't see much in debuggers for some reason. > > > There is extra allowance in the m3core change such > that users of fork + do work (as opposed to fork + exec) > may or may not call ForkAll, depending on if they > feel a need for their own threads to be recreated, > and if they've kept track of how to recreate them, > or just rely on the runtime to know all the threads. > > > There are three runtime threads that are sometimes > created in the parent, and if so, recreated in the child. > background collector, foreground collector, weak ref thread > > > I'll try to poke at it some more. > > > I'm not sure what is the best way to suspend all threads. > I tried a few differnt ways. > SuspendOthers > LockHeap > pthread_mutex_lock > various combinations > > > It is deliberate that pthread specific code is in common/Thread.i3. > That way code can be portable, at least among the two Posix thread implementations. > > > - Jay > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:31 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Can you sketch the approach you've taken? > > > On 17 Mar 2010, at 11:39, Jay K wrote: > > I have something working on Solaris now. > More details after testing on Linux and Darwin. > > - Jay > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Exec what? > You'd have to change the code to carefully reach the same place. > > - Jay > > Subject: Re: [M3devel] fork/cvsup > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 09:28:14 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not just exec in the child? > > On 17 Mar 2010, at 03:47, Jay K wrote: > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such time as one of the exec functions is > called. All APIs, including global data symbols, in any framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 23:05:25 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 22:05:25 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu> References: , , ,,, , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu> Message-ID: Tony, I don't know. Here is some "argument', but I'm not sure. Adding threads does something different. Such threads would share mutation to global state. I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:30:47 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 14:13, Jay K wrote: --- bad news: It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. Restart manually. Works again. Around 9 times. Then hangs again. That is on Linux/x86 and Solaris/sparc. Doesn't work at all on Mac/amd64, just hangs. --- sketch: m3core uses pthread_atfork to selectively reinitialize Mainly to only have one thread. common Thread.PThreadAtFork is provided for others to do the same It is deliberately in a portable interface. Thread.ReforkThreadAfterProcessFork Is provided for users to restart threads from their child AtFork hander. This is used by the allocator/collector. Thread.ForkProcessAndAllThreads() Is used by "lazy" clients who want to restart all their threads but didn't keep track of them. The runtime can do it for them. This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads or not, depending on if they need their threads restarted. The runtime takes care of its threads either way. --- What'd I'd written up: attached works typically 9 times on Linux and Solaris before server hangs again. No improvement on Darwin, just hangs. Can't see much in debuggers for some reason. There is extra allowance in the m3core change such that users of fork + do work (as opposed to fork + exec) may or may not call ForkAll, depending on if they feel a need for their own threads to be recreated, and if they've kept track of how to recreate them, or just rely on the runtime to know all the threads. There are three runtime threads that are sometimes created in the parent, and if so, recreated in the child. background collector, foreground collector, weak ref thread I'll try to poke at it some more. I'm not sure what is the best way to suspend all threads. I tried a few differnt ways. SuspendOthers LockHeap pthread_mutex_lock various combinations It is deliberate that pthread specific code is in common/Thread.i3. That way code can be portable, at least among the two Posix thread implementations. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:31 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 23:18:37 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 22:18:37 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: References: , ,,,,,,, , , ,,, , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, Message-ID: Maybe an easier approach is to offer a way in quake to select user threads, that then "sticks" automatically through runtime? That either implies build_standalone, or provides a differently named m3core.so? Replacing m3core.so doesn't work, and wouldn't solve this problem. Presumably one would want cvsup to coexist with non-user threads programs. You have expressed not liking these ideas in the past though. It wouldn't necessarily cost even a function pointer indirection for pthreads. That is, it wouldn't necessarily be an @M3userthreads runtime choice. The choice would be made at compile time and the appropriate functions called directly. Well, granted, the easiest way to implement that is "directly" through other functions. Programs that need "fork all" semantics could use user threads. We could alter the types slightly I think so they have the same fingerprints. So that everything wouldn't have to be recompiled, so that it'd be a build time replacement of one .lib or the other. Ie. first fix it so that swapping .so files works. And then make user_threads imply build_standalone. Does that make sense? To repeat: first adjust the two Thread.T to match, if that is possible. Not their code, just the types. And then build m3core twice. And have user threads imply build_standalone. It'd probably just be a matter of adding a few unused fields. I think I'll first try the "limited atfork" idea. That code is in hand already at least to start. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 22:05:25 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Tony, I don't know. Here is some "argument', but I'm not sure. Adding threads does something different. Such threads would share mutation to global state. I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:30:47 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 14:13, Jay K wrote: --- bad news: It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. Restart manually. Works again. Around 9 times. Then hangs again. That is on Linux/x86 and Solaris/sparc. Doesn't work at all on Mac/amd64, just hangs. --- sketch: m3core uses pthread_atfork to selectively reinitialize Mainly to only have one thread. common Thread.PThreadAtFork is provided for others to do the same It is deliberately in a portable interface. Thread.ReforkThreadAfterProcessFork Is provided for users to restart threads from their child AtFork hander. This is used by the allocator/collector. Thread.ForkProcessAndAllThreads() Is used by "lazy" clients who want to restart all their threads but didn't keep track of them. The runtime can do it for them. This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads or not, depending on if they need their threads restarted. The runtime takes care of its threads either way. --- What'd I'd written up: attached works typically 9 times on Linux and Solaris before server hangs again. No improvement on Darwin, just hangs. Can't see much in debuggers for some reason. There is extra allowance in the m3core change such that users of fork + do work (as opposed to fork + exec) may or may not call ForkAll, depending on if they feel a need for their own threads to be recreated, and if they've kept track of how to recreate them, or just rely on the runtime to know all the threads. There are three runtime threads that are sometimes created in the parent, and if so, recreated in the child. background collector, foreground collector, weak ref thread I'll try to poke at it some more. I'm not sure what is the best way to suspend all threads. I tried a few differnt ways. SuspendOthers LockHeap pthread_mutex_lock various combinations It is deliberate that pthread specific code is in common/Thread.i3. That way code can be portable, at least among the two Posix thread implementations. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:31 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 17 23:32:44 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 17 Mar 2010 18:32:44 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , , , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu> Message-ID: <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> What are the expectations in the cvsup child regarding the threads it inherits? On 17 Mar 2010, at 18:05, Jay K wrote: > Tony, I don't know. > Here is some "argument', but I'm not sure. > > > Adding threads does something different. Such threads would share mutation to global state. > I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. > > I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave > people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. > > > There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:30:47 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 17 Mar 2010, at 14:13, Jay K wrote: > > --- > bad news: > It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. > Restart manually. Works again. Around 9 times. Then hangs again. > That is on Linux/x86 and Solaris/sparc. > Doesn't work at all on Mac/amd64, just hangs. > > --- > sketch: > m3core uses pthread_atfork to selectively reinitialize > Mainly to only have one thread. > > > common Thread.PThreadAtFork is provided for others to do the same > It is deliberately in a portable interface. > > > Thread.ReforkThreadAfterProcessFork > Is provided for users to restart threads from their child AtFork hander. > This is used by the allocator/collector. > > > Thread.ForkProcessAndAllThreads() > Is used by "lazy" clients who want to restart all their threads > but didn't keep track of them. The runtime can do it for them. > > > This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads > or not, depending on if they need their threads restarted. > The runtime takes care of its threads either way. > > > --- > What'd I'd written up: > > attached works typically 9 times on Linux and Solaris > before server hangs again. > > > No improvement on Darwin, just hangs. > Can't see much in debuggers for some reason. > > > There is extra allowance in the m3core change such > that users of fork + do work (as opposed to fork + exec) > may or may not call ForkAll, depending on if they > feel a need for their own threads to be recreated, > and if they've kept track of how to recreate them, > or just rely on the runtime to know all the threads. > > > There are three runtime threads that are sometimes > created in the parent, and if so, recreated in the child. > background collector, foreground collector, weak ref thread > > > I'll try to poke at it some more. > > > I'm not sure what is the best way to suspend all threads. > I tried a few differnt ways. > SuspendOthers > LockHeap > pthread_mutex_lock > various combinations > > > It is deliberate that pthread specific code is in common/Thread.i3. > That way code can be portable, at least among the two Posix thread implementations. > > > - Jay > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:31 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Can you sketch the approach you've taken? > > > On 17 Mar 2010, at 11:39, Jay K wrote: > > I have something working on Solaris now. > More details after testing on Linux and Darwin. > > - Jay > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Exec what? > You'd have to change the code to carefully reach the same place. > > - Jay > > Subject: Re: [M3devel] fork/cvsup > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 09:28:14 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not just exec in the child? > > On 17 Mar 2010, at 03:47, Jay K wrote: > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such time as one of the exec functions is > called. All APIs, including global data symbols, in any framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 17 23:40:17 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 17 Mar 2010 18:40:17 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , , , , , , , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, Message-ID: <5EBE34A1-B22E-4E03-AB59-5A7E4ABFAAA9@cs.purdue.edu> That makes little sense. Surely the cvsup server would want to run with proper multi-threading? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 18:18, Jay K wrote: > Maybe an easier approach is to offer a way in quake to select user threads, > that then "sticks" automatically through runtime? > That either implies build_standalone, or provides a differently named m3core.so? > Replacing m3core.so doesn't work, and wouldn't solve this problem. Presumably > one would want cvsup to coexist with non-user threads programs. > > > You have expressed not liking these ideas in the past though. > > > It wouldn't necessarily cost even a function pointer indirection for pthreads. > That is, it wouldn't necessarily be an @M3userthreads runtime choice. > > > The choice would be made at compile time and the appropriate functions called directly. > Well, granted, the easiest way to implement that is "directly" through other functions. > > > Programs that need "fork all" semantics could use user threads. > > > We could alter the types slightly I think so they have the same fingerprints. > So that everything wouldn't have to be recompiled, so that it'd be a build time > replacement of one .lib or the other. Ie. first fix it so that swapping .so files > works. And then make user_threads imply build_standalone. > Does that make sense? > To repeat: first adjust the two Thread.T to match, if that is possible. > Not their code, just the types. > And then build m3core twice. > And have user threads imply build_standalone. > > > It'd probably just be a matter of adding a few unused fields. > > > I think I'll first try the "limited atfork" idea. > That code is in hand already at least to start. > > > - Jay > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 22:05:25 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Tony, I don't know. > Here is some "argument', but I'm not sure. > > > Adding threads does something different. Such threads would share mutation to global state. > I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. > > I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave > people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. > > > There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:30:47 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 17 Mar 2010, at 14:13, Jay K wrote: > > --- > bad news: > It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. > Restart manually. Works again. Around 9 times. Then hangs again. > That is on Linux/x86 and Solaris/sparc. > Doesn't work at all on Mac/amd64, just hangs. > > --- > sketch: > m3core uses pthread_atfork to selectively reinitialize > Mainly to only have one thread. > > > common Thread.PThreadAtFork is provided for others to do the same > It is deliberately in a portable interface. > > > Thread.ReforkThreadAfterProcessFork > Is provided for users to restart threads from their child AtFork hander. > This is used by the allocator/collector. > > > Thread.ForkProcessAndAllThreads() > Is used by "lazy" clients who want to restart all their threads > but didn't keep track of them. The runtime can do it for them. > > > This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads > or not, depending on if they need their threads restarted. > The runtime takes care of its threads either way. > > > --- > What'd I'd written up: > > attached works typically 9 times on Linux and Solaris > before server hangs again. > > > No improvement on Darwin, just hangs. > Can't see much in debuggers for some reason. > > > There is extra allowance in the m3core change such > that users of fork + do work (as opposed to fork + exec) > may or may not call ForkAll, depending on if they > feel a need for their own threads to be recreated, > and if they've kept track of how to recreate them, > or just rely on the runtime to know all the threads. > > > There are three runtime threads that are sometimes > created in the parent, and if so, recreated in the child. > background collector, foreground collector, weak ref thread > > > I'll try to poke at it some more. > > > I'm not sure what is the best way to suspend all threads. > I tried a few differnt ways. > SuspendOthers > LockHeap > pthread_mutex_lock > various combinations > > > It is deliberate that pthread specific code is in common/Thread.i3. > That way code can be portable, at least among the two Posix thread implementations. > > > - Jay > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:31 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Can you sketch the approach you've taken? > > > On 17 Mar 2010, at 11:39, Jay K wrote: > > I have something working on Solaris now. > More details after testing on Linux and Darwin. > > - Jay > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Exec what? > You'd have to change the code to carefully reach the same place. > > - Jay > > Subject: Re: [M3devel] fork/cvsup > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 09:28:14 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not just exec in the child? > > On 17 Mar 2010, at 03:47, Jay K wrote: > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such time as one of the exec functions is > called. All APIs, including global data symbols, in any framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 18 00:04:23 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 23:04:23 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: <5EBE34A1-B22E-4E03-AB59-5A7E4ABFAAA9@cs.purdue.edu> References: , , , , ,,, , , ,,, , ,,, , ,,<553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, ,,, , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, , , , <5EBE34A1-B22E-4E03-AB59-5A7E4ABFAAA9@cs.purdue.edu> Message-ID: [somewhat self follow up] These ideas are not mutually exclusive. It should be easier to switch to user threads. One might imagine "IMPORT UserThread AS Thread" though with that construct, you'd have a system using a mix of each thread type, depending on what a module imported, on a module by module basis. Anyway, let me try something that doesn't involve user threads. Something like what I sent but smaller. I also need to see if user threads exhibit the "only it works 9 times" problem, and if I can debug that either way. It really seems to be a fixed number of times. Hm. I have been prone to say -C 99. I should make sure that isn't interpreted as -C 9. I do sometimes say -C 2. And that's supposed to be sequential clients and I've seen what it does when it hits the limit it prints a specific message (some forms do hit the limit, because the hangs are in child servers and the master server knows they are still running). Hm. I'll have to dig in more/again. I'm still not comfortable with my use of SuspendOthers. Is it guaranteed where such threads are? I know they are sitting in the signal header, on some platforms, but no guarantee what got interrupted by it, right? They don't have any of the Thread.m3 locks? But they might have some other locks? I guess like the atfork documentation says, for this stuff to work, everyone with a lock has to use atfork. Good point about the server, though it never historically has. We're stuck between user threads used to work for it and kernel threads probably never have. Or it could be my theories are all wrong. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 18:40:17 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup That makes little sense. Surely the cvsup server would want to run with proper multi-threading? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 18:18, Jay K wrote: Maybe an easier approach is to offer a way in quake to select user threads, that then "sticks" automatically through runtime? That either implies build_standalone, or provides a differently named m3core.so? Replacing m3core.so doesn't work, and wouldn't solve this problem. Presumably one would want cvsup to coexist with non-user threads programs. You have expressed not liking these ideas in the past though. It wouldn't necessarily cost even a function pointer indirection for pthreads. That is, it wouldn't necessarily be an @M3userthreads runtime choice. The choice would be made at compile time and the appropriate functions called directly. Well, granted, the easiest way to implement that is "directly" through other functions. Programs that need "fork all" semantics could use user threads. We could alter the types slightly I think so they have the same fingerprints. So that everything wouldn't have to be recompiled, so that it'd be a build time replacement of one .lib or the other. Ie. first fix it so that swapping .so files works. And then make user_threads imply build_standalone. Does that make sense? To repeat: first adjust the two Thread.T to match, if that is possible. Not their code, just the types. And then build m3core twice. And have user threads imply build_standalone. It'd probably just be a matter of adding a few unused fields. I think I'll first try the "limited atfork" idea. That code is in hand already at least to start. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 22:05:25 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Tony, I don't know. Here is some "argument', but I'm not sure. Adding threads does something different. Such threads would share mutation to global state. I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:30:47 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 14:13, Jay K wrote: --- bad news: It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. Restart manually. Works again. Around 9 times. Then hangs again. That is on Linux/x86 and Solaris/sparc. Doesn't work at all on Mac/amd64, just hangs. --- sketch: m3core uses pthread_atfork to selectively reinitialize Mainly to only have one thread. common Thread.PThreadAtFork is provided for others to do the same It is deliberately in a portable interface. Thread.ReforkThreadAfterProcessFork Is provided for users to restart threads from their child AtFork hander. This is used by the allocator/collector. Thread.ForkProcessAndAllThreads() Is used by "lazy" clients who want to restart all their threads but didn't keep track of them. The runtime can do it for them. This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads or not, depending on if they need their threads restarted. The runtime takes care of its threads either way. --- What'd I'd written up: attached works typically 9 times on Linux and Solaris before server hangs again. No improvement on Darwin, just hangs. Can't see much in debuggers for some reason. There is extra allowance in the m3core change such that users of fork + do work (as opposed to fork + exec) may or may not call ForkAll, depending on if they feel a need for their own threads to be recreated, and if they've kept track of how to recreate them, or just rely on the runtime to know all the threads. There are three runtime threads that are sometimes created in the parent, and if so, recreated in the child. background collector, foreground collector, weak ref thread I'll try to poke at it some more. I'm not sure what is the best way to suspend all threads. I tried a few differnt ways. SuspendOthers LockHeap pthread_mutex_lock various combinations It is deliberate that pthread specific code is in common/Thread.i3. That way code can be portable, at least among the two Posix thread implementations. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:31 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 18 00:12:09 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 23:12:09 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> References: , , , ,,, , ,,, ,,, , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, , <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> Message-ID: I don't know. I know there is one thread that it merely shuts right down in the child. It does that by queueing a message to it. The initial problem you see in the current code is it hangs trying to do that. You can basically just remove that code (except then it won't work with user threads probably!). The problem you hit after that is ThreadPThread.m3 getting, I forget the errno, but pthread_kill complains that you give it a nonexistant thread. That's presumably because the child process has inherited the parent's data as to existant threads. "limited atfork" addresses that. "limited atfork" means, a subset of the diff I sent. So the next things for me to try: - verify user threads doesn't fail after 9 also - verify that 9 isn't associated with "-C 99". - assuming no to both of those, try "limited atfork" and remove the code to shutdown the (nonexistant) dispatcher thread. If that works, almost done. Only remaining part would be to expose a boolean from Thread.i3 so cvsup could make the right choice. There might be a way to structure the cvsup code to work either way and not have to know. Something like signaling the thread ahead of time that it might be going away, and unsignaling only in the parent. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 18:32:44 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup What are the expectations in the cvsup child regarding the threads it inherits? On 17 Mar 2010, at 18:05, Jay K wrote: Tony, I don't know. Here is some "argument', but I'm not sure. Adding threads does something different. Such threads would share mutation to global state. I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:30:47 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 14:13, Jay K wrote: --- bad news: It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. Restart manually. Works again. Around 9 times. Then hangs again. That is on Linux/x86 and Solaris/sparc. Doesn't work at all on Mac/amd64, just hangs. --- sketch: m3core uses pthread_atfork to selectively reinitialize Mainly to only have one thread. common Thread.PThreadAtFork is provided for others to do the same It is deliberately in a portable interface. Thread.ReforkThreadAfterProcessFork Is provided for users to restart threads from their child AtFork hander. This is used by the allocator/collector. Thread.ForkProcessAndAllThreads() Is used by "lazy" clients who want to restart all their threads but didn't keep track of them. The runtime can do it for them. This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads or not, depending on if they need their threads restarted. The runtime takes care of its threads either way. --- What'd I'd written up: attached works typically 9 times on Linux and Solaris before server hangs again. No improvement on Darwin, just hangs. Can't see much in debuggers for some reason. There is extra allowance in the m3core change such that users of fork + do work (as opposed to fork + exec) may or may not call ForkAll, depending on if they feel a need for their own threads to be recreated, and if they've kept track of how to recreate them, or just rely on the runtime to know all the threads. There are three runtime threads that are sometimes created in the parent, and if so, recreated in the child. background collector, foreground collector, weak ref thread I'll try to poke at it some more. I'm not sure what is the best way to suspend all threads. I tried a few differnt ways. SuspendOthers LockHeap pthread_mutex_lock various combinations It is deliberate that pthread specific code is in common/Thread.i3. That way code can be portable, at least among the two Posix thread implementations. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:31 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Wed Mar 17 13:29:43 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Wed, 17 Mar 2010 13:29:43 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> Message-ID: <1268828983.2906.0.camel@faramir.m3w.org> Yes, I can. I am building it regularly, from version to version, at least few times a year. And it also worked with my ancient kernel thread implementation. Was one of stress tests. On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > (Can anyone claim to have seen cvsup server work with kernel threads?) -- Dragi?a Duri? From jay.krell at cornell.edu Thu Mar 18 11:09:38 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 18 Mar 2010 10:09:38 +0000 Subject: [M3devel] cvsup In-Reply-To: <1268828983.2906.0.camel@faramir.m3w.org> References: ,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , <1268828983.2906.0.camel@faramir.m3w.org> Message-ID: Dragisha, Really? The server? With the -C flag and/or -f? Evidence is very very very very very good as to what is going on here. It is all related to "fork1" vs. "forkall". fork1 is the overwhelming usual behavior (Solaris has an option), and basically *any* library that uses pthreads is obligated to use pthread_atfork, be it a C library or the Modula-3 library, etc.. The application cannot do things, unless the library exposes its internal locks. The Posix spec for pthread_atfork explains the problem. Consider C code that links in Modula-3 code. C code is fairly free to use the fork + do work model. It isn't very common, but it does exist, and people have gone to extra length to keep it viable. bash and ccache I believe both use this model. Granted, if you had your own kernel thread implementation, it might have addressed this. I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions. The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork. I want to still try out the "bigger" change, but with the simpler lock/unlock. And test on Solaris and Darwin. I think for SuspendOthers to not work is not surprising. The threads can still hold a few locks even if suspended, I'm pretty sure. The point is to fork in a "controlled" fashion -- all locks held, and in the right order, so that both parent and child can release them. Taking "all" the locks in Prepare is more reliable. I do wonder if really "all" locks need to be taken. I only take the static ones declared in PThreadThread.c. - Jay > Subject: Re: [M3devel] cvsup > From: dragisha at m3w.org > To: jay.krell at cornell.edu > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > Yes, I can. I am building it regularly, from version to version, at > least few times a year. And it also worked with my ancient kernel thread > implementation. Was one of stress tests. > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Mar 18 13:56:11 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 18 Mar 2010 13:56:11 +0100 Subject: [M3devel] cvsup In-Reply-To: References: ,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> , ,<1268828983.2906.0.camel@faramir.m3w.org> Message-ID: <1268916971.2586.3.camel@faramir.m3w.org> Client, it's what I am using. And it's dead as we speak. Last worked before I pulled/built RC4 On Thu, 2010-03-18 at 10:09 +0000, Jay K wrote: > Dragisha, Really? The server? With the -C flag and/or -f? > > Evidence is very very very very very good as to what is going on here. > It is all related to "fork1" vs. "forkall". > fork1 is the overwhelming usual behavior (Solaris has an option), > and basically *any* library that uses pthreads is obligated to use > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > The application cannot do things, unless > the library exposes its internal locks. The Posix spec for > pthread_atfork explains the problem. > > Consider C code that links in Modula-3 code. > C code is fairly free to use the fork + do work model. It isn't very > common, > but it does exist, and people have gone to extra length to keep it > viable. > bash and ccache I believe both use this model. > > > Granted, if you had your own kernel thread implementation, it might > have addressed this. > > > I have a version now that has served over 1000 requests, similar to > what I sent, but a bit reduced. The main change was I just called > pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead > of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just > Linux. This version also doesn't expose the ForkProcessAndAllThreads > functions. > The client has to restart its threads, or in the cvsupd case, don't > depend on the dispatcher thread to survive fork. > I want to still try out the "bigger" change, but with the simpler > lock/unlock. > And test on Solaris and Darwin. > > > I think for SuspendOthers to not work is not surprising. > The threads can still hold a few locks even if suspended, I'm pretty > sure. > The point is to fork in a "controlled" fashion -- all locks held, and > in > the right order, so that both parent and child can release them. > > > Taking "all" the locks in Prepare is more reliable. > > > I do wonder if really "all" locks need to be taken. > I only take the static ones declared in PThreadThread.c. > > - Jay > > > > Subject: Re: [M3devel] cvsup > > From: dragisha at m3w.org > > To: jay.krell at cornell.edu > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > Yes, I can. I am building it regularly, from version to version, at > > least few times a year. And it also worked with my ancient kernel > thread > > implementation. Was one of stress tests. > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > (Can anyone claim to have seen cvsup server work with kernel > threads?) > > -- > > Dragi?a Duri? > > -- Dragi?a Duri? From wagner at elegosoft.com Thu Mar 18 15:25:10 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Thu, 18 Mar 2010 15:25:10 +0100 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , ,,, , ,,, ,,, , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, , <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> Message-ID: <20100318152510.nr94m5wi1wkw0048@mail.elegosoft.com> I'm not able to follow your discussion completely now, but some information about CVSup can be found at http://www.cvsup.org/ http://www.cvsup.org/howsofast.html AFAIK it uses the traditional Unix daemon pattern to fork a new server process for each client connection, which then creates threads for the streaming protocol. It should be possible to change the pattern to use threads only, but I'd rather like it if we could support the traditional Unix daemon pattern in a general form, too. Modula-3 is a systems programming language after all. Olaf Quoting Jay K : > > I don't know. I know there is one thread that it merely shuts right > down in the child. > > It does that by queueing a message to it. > > The initial problem you see in the current code is it hangs trying > to do that. > > > > > > You can basically just remove that code (except then it won't work > > with user threads probably!). > > > > > > The problem you hit after that is ThreadPThread.m3 getting, > > I forget the errno, but pthread_kill complains that you give > > it a nonexistant thread. That's presumably because the child > > process has inherited the parent's data as to existant threads. > > "limited atfork" addresses that. "limited atfork" means, a subset > > of the diff I sent. > > > > > > So the next things for me to try: > > - verify user threads doesn't fail after 9 also > > - verify that 9 isn't associated with "-C 99". > > - assuming no to both of those, try "limited atfork" and > > remove the code to shutdown the (nonexistant) dispatcher > > thread. If that works, almost done. Only remaining part would > > be to expose a boolean from Thread.i3 so cvsup could > > make the right choice. There might be a way to structure > > the cvsup code to work either way and not have to know. > > Something like signaling the thread ahead of time that it might > > be going away, and unsignaling only in the parent. > > > > > > - Jay > > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 18:32:44 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > What are the expectations in the cvsup child regarding the threads > it inherits? > > > > > On 17 Mar 2010, at 18:05, Jay K wrote: > > Tony, I don't know. > Here is some "argument', but I'm not sure. > > > Adding threads does something different. Such threads would share > mutation to global state. > I'm not a big fan of this model, but fork lets you establish some > perhaps expensive to establish state, then share it cheaply among a > bunch of future threads/processes, that may make their own local > modifications to it. One would have to read the cvsup code a bunch > to determine what it actually does and requires. > > I do suspect there is a general solution. Leaving anyone who uses > platform specific functions to fend for themselves seems a bit > unfair. Which functions to we abtract away in m3ore vs. which do we > leave > people to use on their own? And does that list change much in time? > Well, infinity isn't possible either, granted. And we've only seen > one program so far that cares, we shouldn't spend too much just for > one program. > > > There may be a smaller related fix, where m3core internally uses > atfork, but doesn't expose ForkAll to the client. I know cvsup has > the dispatcher thread that it expects to be inherited by children, > however all it does with it is queue a request to it to shut itself > down. In that way, ForkAll is a waste -- it recreates a thread, only > so the client can shut it down. I can pursue that more. > > > - Jay > > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:30:47 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > I don't think there is a "general" solution to this that should be > applied to the thread library. Modula-3 does not mandate any > support for fork! It is not part of the language. If a program > relies on platform-specific interfaces then it must be the one to > handle situations arising from the problem. Why does cvsup need to > fork in the first place? Surely it can simply add threads to handle > clients as they arrive? > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > On 17 Mar 2010, at 14:13, Jay K wrote: > > --- > bad news: > It doesn't completely work. It works a bunch of times in a row, like > 9, then hangs. > Restart manually. Works again. Around 9 times. Then hangs again. > That is on Linux/x86 and Solaris/sparc. > Doesn't work at all on Mac/amd64, just hangs. > > --- > sketch: > m3core uses pthread_atfork to selectively reinitialize > Mainly to only have one thread. > > > common Thread.PThreadAtFork is provided for others to do the same > It is deliberately in a portable interface. > > > Thread.ReforkThreadAfterProcessFork > Is provided for users to restart threads from their child AtFork hander. > This is used by the allocator/collector. > > > Thread.ForkProcessAndAllThreads() > Is used by "lazy" clients who want to restart all their threads > but didn't keep track of them. The runtime can do it for them. > > > This allows for "fork + do work" folks do call or not call > ForkProcessAndAllThreads > or not, depending on if they need their threads restarted. > The runtime takes care of its threads either way. > > > --- > What'd I'd written up: > > attached works typically 9 times on Linux and Solaris > before server hangs again. > > > No improvement on Darwin, just hangs. > Can't see much in debuggers for some reason. > > > There is extra allowance in the m3core change such > that users of fork + do work (as opposed to fork + exec) > may or may not call ForkAll, depending on if they > feel a need for their own threads to be recreated, > and if they've kept track of how to recreate them, > or just rely on the runtime to know all the threads. > > > There are three runtime threads that are sometimes > created in the parent, and if so, recreated in the child. > background collector, foreground collector, weak ref thread > > > I'll try to poke at it some more. > > > I'm not sure what is the best way to suspend all threads. > I tried a few differnt ways. > SuspendOthers > LockHeap > pthread_mutex_lock > various combinations > > > It is deliberate that pthread specific code is in common/Thread.i3. > That way code can be portable, at least among the two Posix thread > implementations. > > > - Jay > > > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:31 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Can you sketch the approach you've taken? > > > > > On 17 Mar 2010, at 11:39, Jay K wrote: > > I have something working on Solaris now. > More details after testing on Linux and Darwin. > > - Jay > > > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Exec what? > You'd have to change the code to carefully reach the same place. > > - Jay > > > > Subject: Re: [M3devel] fork/cvsup > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 09:28:14 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > > > > Why not just exec in the child? > > > On 17 Mar 2010, at 03:47, Jay K wrote: > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be > totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such > time as one of the exec functions is > called. All APIs, including global data symbols, in any > framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe > or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this > situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute > async-signal-safe operations until such time as one of theexec > functions is called. [THR] Fork handlers may be established by > means of the pthread_atfork() function in order to maintain > application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another > thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the > pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a > record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. > mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be > initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon > creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay > > > > > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From hosking at cs.purdue.edu Thu Mar 18 15:31:15 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 18 Mar 2010 10:31:15 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: <20100318152510.nr94m5wi1wkw0048@mail.elegosoft.com> References: , , , , , , , , , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, , <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> <20100318152510.nr94m5wi1wkw0048@mail.elegosoft.com> Message-ID: <318D6C6F-80BE-41F5-80CC-03387905439B@cs.purdue.edu> Agreed. I do seem to recall cvsupd working with system threads way back when I was testing things. On 18 Mar 2010, at 10:25, Olaf Wagner wrote: > I'm not able to follow your discussion completely now, but some > information about CVSup can be found at > > http://www.cvsup.org/ > http://www.cvsup.org/howsofast.html > > AFAIK it uses the traditional Unix daemon pattern to fork a new > server process for each client connection, which then creates threads > for the streaming protocol. > > It should be possible to change the pattern to use threads only, > but I'd rather like it if we could support the traditional Unix > daemon pattern in a general form, too. Modula-3 is a systems > programming language after all. > > Olaf > > Quoting Jay K : > >> >> I don't know. I know there is one thread that it merely shuts right down in the child. >> >> It does that by queueing a message to it. >> >> The initial problem you see in the current code is it hangs trying to do that. >> >> >> >> >> >> You can basically just remove that code (except then it won't work >> >> with user threads probably!). >> >> >> >> >> >> The problem you hit after that is ThreadPThread.m3 getting, >> >> I forget the errno, but pthread_kill complains that you give >> >> it a nonexistant thread. That's presumably because the child >> >> process has inherited the parent's data as to existant threads. >> >> "limited atfork" addresses that. "limited atfork" means, a subset >> >> of the diff I sent. >> >> >> >> >> >> So the next things for me to try: >> >> - verify user threads doesn't fail after 9 also >> >> - verify that 9 isn't associated with "-C 99". >> >> - assuming no to both of those, try "limited atfork" and >> >> remove the code to shutdown the (nonexistant) dispatcher >> >> thread. If that works, almost done. Only remaining part would >> >> be to expose a boolean from Thread.i3 so cvsup could >> >> make the right choice. There might be a way to structure >> >> the cvsup code to work either way and not have to know. >> >> Something like signaling the thread ahead of time that it might >> >> be going away, and unsignaling only in the parent. >> >> >> >> >> >> - Jay >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 18:32:44 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> What are the expectations in the cvsup child regarding the threads it inherits? >> >> >> >> >> On 17 Mar 2010, at 18:05, Jay K wrote: >> >> Tony, I don't know. >> Here is some "argument', but I'm not sure. >> >> >> Adding threads does something different. Such threads would share mutation to global state. >> I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. >> >> I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave >> people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. >> >> >> There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. >> >> >> - Jay >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:30:47 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? >> >> >> >> >> Antony Hosking | Associate Professor | Computer Science | Purdue University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> On 17 Mar 2010, at 14:13, Jay K wrote: >> >> --- >> bad news: >> It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. >> Restart manually. Works again. Around 9 times. Then hangs again. >> That is on Linux/x86 and Solaris/sparc. >> Doesn't work at all on Mac/amd64, just hangs. >> >> --- >> sketch: >> m3core uses pthread_atfork to selectively reinitialize >> Mainly to only have one thread. >> >> >> common Thread.PThreadAtFork is provided for others to do the same >> It is deliberately in a portable interface. >> >> >> Thread.ReforkThreadAfterProcessFork >> Is provided for users to restart threads from their child AtFork hander. >> This is used by the allocator/collector. >> >> >> Thread.ForkProcessAndAllThreads() >> Is used by "lazy" clients who want to restart all their threads >> but didn't keep track of them. The runtime can do it for them. >> >> >> This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads >> or not, depending on if they need their threads restarted. >> The runtime takes care of its threads either way. >> >> >> --- >> What'd I'd written up: >> >> attached works typically 9 times on Linux and Solaris >> before server hangs again. >> >> >> No improvement on Darwin, just hangs. >> Can't see much in debuggers for some reason. >> >> >> There is extra allowance in the m3core change such >> that users of fork + do work (as opposed to fork + exec) >> may or may not call ForkAll, depending on if they >> feel a need for their own threads to be recreated, >> and if they've kept track of how to recreate them, >> or just rely on the runtime to know all the threads. >> >> >> There are three runtime threads that are sometimes >> created in the parent, and if so, recreated in the child. >> background collector, foreground collector, weak ref thread >> >> >> I'll try to poke at it some more. >> >> >> I'm not sure what is the best way to suspend all threads. >> I tried a few differnt ways. >> SuspendOthers >> LockHeap >> pthread_mutex_lock >> various combinations >> >> >> It is deliberate that pthread specific code is in common/Thread.i3. >> That way code can be portable, at least among the two Posix thread implementations. >> >> >> - Jay >> >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:01:31 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> Can you sketch the approach you've taken? >> >> >> >> >> On 17 Mar 2010, at 11:39, Jay K wrote: >> >> I have something working on Solaris now. >> More details after testing on Linux and Darwin. >> >> - Jay >> >> >> >> From: jay.krell at cornell.edu >> To: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:01:15 +0000 >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> Exec what? >> You'd have to change the code to carefully reach the same place. >> >> - Jay >> >> >> >> Subject: Re: [M3devel] fork/cvsup >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 09:28:14 -0400 >> CC: m3devel at elegosoft.com >> To: jay.krell at cornell.edu >> >> >> >> >> Why not just exec in the child? >> >> >> On 17 Mar 2010, at 03:47, Jay K wrote: >> >> http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html >> >> >> There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself >> self to only executing async-signal safe operations until such time as one of the exec functions is >> called. All APIs, including global data symbols, in any framework or library should be assumed to be >> unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use >> these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. >> self. >> >> >> http://www.opengroup.org/onlinepubs/000095399/functions/fork.html >> >> Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. >> >> >> I've run through a few theories so far. >> Current thinking is related to what Tony said: >> use pthread_atfork: >> in prepare, stopworld >> in parent, resumeworld >> You don't want the child to be mid-gc for example, on another thread. Or mid-anything. >> in child, reinitialize -- current thread is the only thread >> >> >> Also in the cvsup code, ShutDown should just call DoShutDown immediately. >> I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. >> pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. >> >> >> I don't have a theory as to why user threads work. >> >> >> I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). >> I was expecting more cases to act like mmap(shared), but none did, only it. >> >> >> I experimented with having mutexes and condition variables be initialize up front instead of on-demand. >> Via changing cvsup to lock/unlock or broadcast immediately upon creating them. >> On the theory that might let them work across process. >> That didn't make a difference. >> >> >> - Jay >> >> >> >> >> > > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > From jay.krell at cornell.edu Thu Mar 18 16:45:11 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 18 Mar 2010 15:45:11 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <1268828983.2906.0.camel@faramir.m3w.org>, Message-ID: To repeat myself: > and basically *any* library that uses pthreads is obligated to use > pthread_atfork, be it a C library or the Modula-3 library, etc.. This is the crux of the matter. We are being "bad pthread citizens" by not using pthread_atfork. Granted, we are probably in large company. There is a smaller fix and a larger fix, but it seems very clear we should take one of them. The small fix is: common/Thread.i3: add PThreadAtFork It does nothing on posix/nt. It calls pthread_atfork on pthread. In RTCollector.m3 give a child handler that stores FALSE in the three booleans that indicate the three threads have started. No prepare or parent handler I believe. Though I should check for global locks there. Probably the locks in ThreadPThread suffice? In ThreadPThread.m3 provide three handlers: prepare: lock "everything", at least the 5 static locks. I might try walking all the threads and locking them too. parent: unlock everything child: unlock everything and reinitialize the globals Note that the atfork handlers run in many more programs than cvsup. They would be used also with fork + exec. Perhaps a way to disable that -- no way in Posix, we could just provide a boolean to ourselves. The larger fix would be to provide a common/Thread.i3 function to call fork() *and* recreate all Modula-3 threads in the child process. That is what I sent out. In the second case, you also then need to allow that the caller may or may not use that function, so you still need atfork handlers, but they have to be slightly different, as the diffs I sent show. I believe strongly this is the way to go. It is how one is a "good pthread citizen". Now, granted, I don't think most libraries are. Witness the caveat about fork() in the Darwin manpage and I think even the Posix spec. But pthread_atfork is meant to solve this. I don't suggest a full audit and add atfork handlers everywhere. But m3core we can at least do, and that should satisfy cvsup. Should check libm3 too. Plus, as long as each module takes reponsibility for itself, it shouldn't be so hard. That is, I don't think the case is all that strong for providing the "forkall" function that is in the diffs I sent. I'll do more testing and send out diffs "later". It could be as long as a week, I have a bunch of things to do. Or maybe just a day. :) I tested a form of this fix + cvsup + user threads and it appears that cvsup can do one small thing and thereby work either way, without knowledge as to the way. That is, it can shutdown the dispatcher "directly", instead of queuing to it. I'm not even sure the dispatcher thread is really needed. As I said, I don't believe the application can handle this all itself. Any library that uses pthreads is obligated to play into the general mechanism. Generally the internals are not exposed for the application to do it. Treating Modula-3 as a closed system in which nobody uses anything outside of or "below" m3core/libm3 I don't think is practical. "applications", arguably defines as "processes", are often composed of fairly disparate bodies of code that don't necessarily expose much to each other. For example, they might each create their own worker threads and not expose them. This is what pthread_atfork is for. Actually Tony, you gave me the lead here. :) There is a problem that m3posixthreads (user) and pthreads semantics diverage, and..I haven't been able to think of an way to fix that. Well, it is actually easy to make "forkall" the default and only behavior actually, on user and pthreads (but not NT). But I don't see a way to make "fork1" the behavior for user threads. You can associate a pid with each thread, and notice when the pid changes, but I don't know which one thread you would keep, and you wouldn't necessarily be able to register pthread_atfork handlers, unless maybe user threads were used only on platforms that actually have pthreads.. And still there's no good way to it on NT I expect. - Jay From: jay.krell at cornell.edu To: dragisha at m3w.org Date: Thu, 18 Mar 2010 10:09:38 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup Dragisha, Really? The server? With the -C flag and/or -f? Evidence is very very very very very good as to what is going on here. It is all related to "fork1" vs. "forkall". fork1 is the overwhelming usual behavior (Solaris has an option), and basically *any* library that uses pthreads is obligated to use pthread_atfork, be it a C library or the Modula-3 library, etc.. The application cannot do things, unless the library exposes its internal locks. The Posix spec for pthread_atfork explains the problem. Consider C code that links in Modula-3 code. C code is fairly free to use the fork + do work model. It isn't very common, but it does exist, and people have gone to extra length to keep it viable. bash and ccache I believe both use this model. Granted, if you had your own kernel thread implementation, it might have addressed this. I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions. The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork. I want to still try out the "bigger" change, but with the simpler lock/unlock. And test on Solaris and Darwin. I think for SuspendOthers to not work is not surprising. The threads can still hold a few locks even if suspended, I'm pretty sure. The point is to fork in a "controlled" fashion -- all locks held, and in the right order, so that both parent and child can release them. Taking "all" the locks in Prepare is more reliable. I do wonder if really "all" locks need to be taken. I only take the static ones declared in PThreadThread.c. - Jay > Subject: Re: [M3devel] cvsup > From: dragisha at m3w.org > To: jay.krell at cornell.edu > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > Yes, I can. I am building it regularly, from version to version, at > least few times a year. And it also worked with my ancient kernel thread > implementation. Was one of stress tests. > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdp at polstra.com Thu Mar 18 17:56:57 2010 From: jdp at polstra.com (John Polstra) Date: Thu, 18 Mar 2010 09:56:57 -0700 Subject: [M3devel] fork/cvsup In-Reply-To: <20100318152510.nr94m5wi1wkw0048@mail.elegosoft.com> References: , , , , , , , , , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, , <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> <20100318152510.nr94m5wi1wkw0048@mail.elegosoft.com> Message-ID: <79503C57-BC05-44E5-B0E2-494639A1FBD7@polstra.com> I don't want to get too involved in this, because it's been years and years since I even glanced at CVSup. (I retired from the software business a couple years ago.) But yes, it forks a new process per client and uses threads for the streaming protocol within each client process. I tried doing it all with threads when I first wrote it, but that turned out to be a bad idea for reasons of robustness. With the all-threads approach, any internal error (assert failure, bounds check, etc.) for any client will kill *all* clients. That's unacceptable from the user's point of view. You are on the right track with this bug. There is some kind of bad interaction between the forks and the threads system. I remember some issues along the same lines when I was developing the software, and I had to find what worked by experiment. At that time there were no standard facilities for using fork within threaded programs. Now that you have switched to using OS-provided threads (a good change, I think), it's not surprising that some problems have cropped up. John On Mar 18, 2010, at 7:25 AM, Olaf Wagner wrote: > I'm not able to follow your discussion completely now, but some > information about CVSup can be found at > > http://www.cvsup.org/ > http://www.cvsup.org/howsofast.html > > AFAIK it uses the traditional Unix daemon pattern to fork a new > server process for each client connection, which then creates threads > for the streaming protocol. > > It should be possible to change the pattern to use threads only, > but I'd rather like it if we could support the traditional Unix > daemon pattern in a general form, too. Modula-3 is a systems > programming language after all. > > Olaf > > Quoting Jay K : > >> >> I don't know. I know there is one thread that it merely shuts >> right down in the child. >> >> It does that by queueing a message to it. >> >> The initial problem you see in the current code is it hangs trying >> to do that. >> >> >> >> >> >> You can basically just remove that code (except then it won't work >> >> with user threads probably!). >> >> >> >> >> >> The problem you hit after that is ThreadPThread.m3 getting, >> >> I forget the errno, but pthread_kill complains that you give >> >> it a nonexistant thread. That's presumably because the child >> >> process has inherited the parent's data as to existant threads. >> >> "limited atfork" addresses that. "limited atfork" means, a subset >> >> of the diff I sent. >> >> >> >> >> >> So the next things for me to try: >> >> - verify user threads doesn't fail after 9 also >> >> - verify that 9 isn't associated with "-C 99". >> >> - assuming no to both of those, try "limited atfork" and >> >> remove the code to shutdown the (nonexistant) dispatcher >> >> thread. If that works, almost done. Only remaining part would >> >> be to expose a boolean from Thread.i3 so cvsup could >> >> make the right choice. There might be a way to structure >> >> the cvsup code to work either way and not have to know. >> >> Something like signaling the thread ahead of time that it might >> >> be going away, and unsignaling only in the parent. >> >> >> >> >> >> - Jay >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 18:32:44 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> What are the expectations in the cvsup child regarding the threads >> it inherits? >> >> >> >> >> On 17 Mar 2010, at 18:05, Jay K wrote: >> >> Tony, I don't know. >> Here is some "argument', but I'm not sure. >> >> >> Adding threads does something different. Such threads would share >> mutation to global state. >> I'm not a big fan of this model, but fork lets you establish some >> perhaps expensive to establish state, then share it cheaply among >> a bunch of future threads/processes, that may make their own >> local modifications to it. One would have to read the cvsup code a >> bunch to determine what it actually does and requires. >> >> I do suspect there is a general solution. Leaving anyone who uses >> platform specific functions to fend for themselves seems a bit >> unfair. Which functions to we abtract away in m3ore vs. which do >> we leave >> people to use on their own? And does that list change much in >> time? Well, infinity isn't possible either, granted. And we've >> only seen one program so far that cares, we shouldn't spend too >> much just for one program. >> >> >> There may be a smaller related fix, where m3core internally uses >> atfork, but doesn't expose ForkAll to the client. I know cvsup has >> the dispatcher thread that it expects to be inherited by children, >> however all it does with it is queue a request to it to shut >> itself down. In that way, ForkAll is a waste -- it recreates a >> thread, only so the client can shut it down. I can pursue that more. >> >> >> - Jay >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:30:47 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> I don't think there is a "general" solution to this that should be >> applied to the thread library. Modula-3 does not mandate any >> support for fork! It is not part of the language. If a program >> relies on platform-specific interfaces then it must be the one to >> handle situations arising from the problem. Why does cvsup need >> to fork in the first place? Surely it can simply add threads to >> handle clients as they arrive? >> >> >> >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> On 17 Mar 2010, at 14:13, Jay K wrote: >> >> --- >> bad news: >> It doesn't completely work. It works a bunch of times in a row, >> like 9, then hangs. >> Restart manually. Works again. Around 9 times. Then hangs again. >> That is on Linux/x86 and Solaris/sparc. >> Doesn't work at all on Mac/amd64, just hangs. >> >> --- >> sketch: >> m3core uses pthread_atfork to selectively reinitialize >> Mainly to only have one thread. >> >> >> common Thread.PThreadAtFork is provided for others to do the same >> It is deliberately in a portable interface. >> >> >> Thread.ReforkThreadAfterProcessFork >> Is provided for users to restart threads from their child AtFork >> hander. >> This is used by the allocator/collector. >> >> >> Thread.ForkProcessAndAllThreads() >> Is used by "lazy" clients who want to restart all their threads >> but didn't keep track of them. The runtime can do it for them. >> >> >> This allows for "fork + do work" folks do call or not call >> ForkProcessAndAllThreads >> or not, depending on if they need their threads restarted. >> The runtime takes care of its threads either way. >> >> >> --- >> What'd I'd written up: >> >> attached works typically 9 times on Linux and Solaris >> before server hangs again. >> >> >> No improvement on Darwin, just hangs. >> Can't see much in debuggers for some reason. >> >> >> There is extra allowance in the m3core change such >> that users of fork + do work (as opposed to fork + exec) >> may or may not call ForkAll, depending on if they >> feel a need for their own threads to be recreated, >> and if they've kept track of how to recreate them, >> or just rely on the runtime to know all the threads. >> >> >> There are three runtime threads that are sometimes >> created in the parent, and if so, recreated in the child. >> background collector, foreground collector, weak ref thread >> >> >> I'll try to poke at it some more. >> >> >> I'm not sure what is the best way to suspend all threads. >> I tried a few differnt ways. >> SuspendOthers >> LockHeap >> pthread_mutex_lock >> various combinations >> >> >> It is deliberate that pthread specific code is in common/Thread.i3. >> That way code can be portable, at least among the two Posix thread >> implementations. >> >> >> - Jay >> >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:01:31 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> Can you sketch the approach you've taken? >> >> >> >> >> On 17 Mar 2010, at 11:39, Jay K wrote: >> >> I have something working on Solaris now. >> More details after testing on Linux and Darwin. >> >> - Jay >> >> >> >> From: jay.krell at cornell.edu >> To: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:01:15 +0000 >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> Exec what? >> You'd have to change the code to carefully reach the same place. >> >> - Jay >> >> >> >> Subject: Re: [M3devel] fork/cvsup >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 09:28:14 -0400 >> CC: m3devel at elegosoft.com >> To: jay.krell at cornell.edu >> >> >> >> >> Why not just exec in the child? >> >> >> On 17 Mar 2010, at 03:47, Jay K wrote: >> >> http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html >> >> >> There are limits to what you can do in the child process. To be >> totally safe you should restrict your-self yourself >> self to only executing async-signal safe operations until such >> time as one of the exec functions is >> called. All APIs, including global data symbols, in any >> framework or library should be assumed to be >> unsafe after a fork() unless explicitly documented to be safe >> or async-signal safe. If you need to use >> these frameworks in the child process, you must exec. In this >> situation it is reasonable to exec your-self. yourself. >> self. >> >> >> http://www.opengroup.org/onlinepubs/000095399/functions/fork.html >> >> Consequently, to avoid errors, the child process may only execute >> async-signal-safe operations until such time as one of theexec >> functions is called. [THR] Fork handlers may be established by >> means of the pthread_atfork() function in order to maintain >> application invariants across fork() calls. >> >> >> I've run through a few theories so far. >> Current thinking is related to what Tony said: >> use pthread_atfork: >> in prepare, stopworld >> in parent, resumeworld >> You don't want the child to be mid-gc for example, on another >> thread. Or mid-anything. >> in child, reinitialize -- current thread is the only thread >> >> >> Also in the cvsup code, ShutDown should just call DoShutDown >> immediately. >> I did that, without m3core changes, and it hits an error in the >> pthread code, signaling a nonexistant thread. >> pthread_atfork/child should address that -- child shouldn't retain >> a record of all the threads in the parent. >> >> >> I don't have a theory as to why user threads work. >> >> >> I experimented with malloc vs. static alloc vs. sbrk vs. >> mmap(private) vs. mmap(shared). >> I was expecting more cases to act like mmap(shared), but none did, >> only it. >> >> >> I experimented with having mutexes and condition variables be >> initialize up front instead of on-demand. >> Via changing cvsup to lock/unlock or broadcast immediately upon >> creating them. >> On the theory that might let them work across process. >> That didn't make a difference. >> >> >> - Jay >> >> >> >> >> > > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, > Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 > 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: > Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: > DE163214194 > From hosking at cs.purdue.edu Thu Mar 18 18:16:03 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 18 Mar 2010 13:16:03 -0400 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <1268828983.2906.0.camel@faramir.m3w.org>, Message-ID: On 18 Mar 2010, at 11:45, Jay K wrote: > To repeat myself: > > > and basically *any* library that uses pthreads is obligated to use > > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > > This is the crux of the matter. > We are being "bad pthread citizens" by not using pthread_atfork. > Granted, we are probably in large company. > > > There is a smaller fix and a larger fix, but it seems very clear we > should take one of them. > > > The small fix is: > common/Thread.i3: add PThreadAtFork Name should probably be something more like Thread.AtFork. > It does nothing on posix/nt. > It calls pthread_atfork on pthread. > > > In RTCollector.m3 give a child handler that stores FALSE in the > three booleans that indicate the three threads have started. > No prepare or parent handler I believe. > Though I should check for global locks there. > Probably the locks in ThreadPThread suffice? I wonder if we should only fork when the collector is turned off? > In ThreadPThread.m3 provide three handlers: > prepare: lock "everything", at least the 5 static locks. > I might try walking all the threads and locking them too. > parent: unlock everything > child: unlock everything and reinitialize the globals > > > Note that the atfork handlers run in many more programs than cvsup. > They would be used also with fork + exec. fork + exec is probably relatively safe already. > Perhaps a way to disable that -- no way in Posix, we could just > provide a boolean to ourselves. Not sure I follow. > The larger fix would be to provide a common/Thread.i3 function > to call fork() *and* recreate all Modula-3 threads in the child process. > That is what I sent out. That's tough to do portably! > In the second case, you also then need to allow that the caller > may or may not use that function, so you still need atfork handlers, > but they have to be slightly different, as the diffs I sent show. > > > I believe strongly this is the way to go. > It is how one is a "good pthread citizen". > Now, granted, I don't think most libraries are. > Witness the caveat about fork() in the Darwin manpage and > I think even the Posix spec. > But pthread_atfork is meant to solve this. > > > I don't suggest a full audit and add atfork handlers everywhere. > But m3core we can at least do, and that should satisfy cvsup. > Should check libm3 too. Really we only need to be concerned with internal threads to the run-time, right? It's up to apps to restart threads in the children as necessary. > Plus, as long as each module takes reponsibility for itself, > it shouldn't be so hard. > That is, I don't think the case is all that strong for providing the "forkall" > function that is in the diffs I sent. Yes, I think that is overkill. > I'll do more testing and send out diffs "later". > It could be as long as a week, I have a bunch of things to do. > Or maybe just a day. :) > > > I tested a form of this fix + cvsup + user threads and it appears > that cvsup can do one small thing and thereby work either way, > without knowledge as to the way. > > > That is, it can shutdown the dispatcher "directly", instead of queuing to it. > I'm not even sure the dispatcher thread is really needed. > > > As I said, I don't believe the application can handle this all itself. > Any library that uses pthreads is obligated to play into the general > mechanism. Generally the internals are not exposed for the application > to do it. > > Treating Modula-3 as a closed system in which nobody uses > anything outside of or "below" m3core/libm3 I don't think is practical. > > > "applications", arguably defines as "processes", are often composed > of fairly disparate bodies of code that don't necessarily expose much > to each other. For example, they might each create their own worker > threads and not expose them. > This is what pthread_atfork is for. > Actually Tony, you gave me the lead here. :) ;-) > > > There is a problem that m3posixthreads (user) and pthreads semantics > diverage, and..I haven't been able to think of an way to fix that. > > > Well, it is actually easy to make "forkall" the default and only behavior actually, > on user and pthreads (but not NT). > > > But I don't see a way to make "fork1" the behavior for user threads. > You can associate a pid with each thread, and notice when the pid changes, > but I don't know which one thread you would keep, and you wouldn't > necessarily be able to register pthread_atfork handlers, unless maybe > user threads were used only on platforms that actually have pthreads.. > > > And still there's no good way to it on NT I expect. > > > - Jay > > From: jay.krell at cornell.edu > To: dragisha at m3w.org > Date: Thu, 18 Mar 2010 10:09:38 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > Dragisha, Really? The server? With the -C flag and/or -f? > > Evidence is very very very very very good as to what is going on here. > It is all related to "fork1" vs. "forkall". > fork1 is the overwhelming usual behavior (Solaris has an option), > and basically *any* library that uses pthreads is obligated to use > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > The application cannot do things, unless > the library exposes its internal locks. The Posix spec for > pthread_atfork explains the problem. > > Consider C code that links in Modula-3 code. > C code is fairly free to use the fork + do work model. It isn't very common, > but it does exist, and people have gone to extra length to keep it viable. > bash and ccache I believe both use this model. > > > Granted, if you had your own kernel thread implementation, it might > have addressed this. > > > I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions. > The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork. > I want to still try out the "bigger" change, but with the simpler lock/unlock. > And test on Solaris and Darwin. > > > I think for SuspendOthers to not work is not surprising. > The threads can still hold a few locks even if suspended, I'm pretty sure. > The point is to fork in a "controlled" fashion -- all locks held, and in > the right order, so that both parent and child can release them. > > > Taking "all" the locks in Prepare is more reliable. > > > I do wonder if really "all" locks need to be taken. > I only take the static ones declared in PThreadThread.c. > > - Jay > > > > Subject: Re: [M3devel] cvsup > > From: dragisha at m3w.org > > To: jay.krell at cornell.edu > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > Yes, I can. I am building it regularly, from version to version, at > > least few times a year. And it also worked with my ancient kernel thread > > implementation. Was one of stress tests. > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > -- > > Dragi?a Duri? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 18 21:55:06 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 18 Mar 2010 20:55:06 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <1268828983.2906.0.camel@faramir.m3w.org>, , Message-ID: > Name should probably be something more like Thread.AtFork. I disagree. It should be PThreadAtFork, because it only does anything when using pthreads. If you are on a non-pthread system (user threads or Win32), I expect, I think, the function won't have any affect. Granted, we could be using user threads on a system that has pthreads, so there is ambiguity, there is ambiguity either way. Does it mean, hey, thin wrapper over pthread_atfork, and I'll call it even if using user threads? Or does it mean, only call it if using pthreads? > I wonder if we should only fork when the collector is turned off? I don't quite follow. You mean, anyone calling fork() must GC.Disable()? Or LockHeap() Or, take all the thread locks? That last point is part of what I'm saying, since the "prepare" callback would do that. pthread_atfork lets you establish preconditions for fork() "distributed" out around the code, including with access to internals. Without having to "coordinate" with the caller of fork(). > fork + exec is probably relatively safe already. Right, but this change "unnecessarily" alters it. The "prepare" callback would still run. You can't discern fork + noexec vs. fork + exec They start the same. I think we are converging on agreement. Let me do some more tuning (stylistic, diff reduction, not perf) and testing. Diffs later, maybe tonight, maybe in a week, depending on life. > Perhaps a way to disable that -- no way in Posix, we could just > provide a boolean to ourselves. > Not sure I follow. I meant, something like, in m3core/libm3 where we have a fork+exec sequence, set a boolean somewhere to tell all of our "prepare" handlers not to waste their time doing anything. fork + exec need not run the prepare handlers. And I worry that all their lock taking might deadlock..but I suspect that would indicate a bug. ? > recreate the threads > That's tough to do portably! Easy with user threads -- automatically happens. Easy with pthreads -- systems with fork(). Hard/impossible on Win32. > Really we only need to be concerned with internal threads > to the run-time, right? It's up to apps to restart threads > in the children as necessary. I can agree with that. It is kind of just a lazy app that says "hey, please recreate all my threads, I didn't keep track of them, so I don't know what to restart, I know you kept track of all of them, so please do it for me". You should be able to "imagine" what my diff looks like. And maybe ok it without seeing it? Anyway, I definitely want to see Darwin not hang first. - Jay Subject: Re: [M3devel] cvsup From: hosking at cs.purdue.edu Date: Thu, 18 Mar 2010 13:16:03 -0400 CC: dragisha at m3w.org; m3devel at elegosoft.com To: jay.krell at cornell.edu On 18 Mar 2010, at 11:45, Jay K wrote: To repeat myself: > and basically *any* library that uses pthreads is obligated to use > pthread_atfork, be it a C library or the Modula-3 library, etc.. This is the crux of the matter. We are being "bad pthread citizens" by not using pthread_atfork. Granted, we are probably in large company. There is a smaller fix and a larger fix, but it seems very clear we should take one of them. The small fix is: common/Thread.i3: add PThreadAtFork Name should probably be something more like Thread.AtFork. It does nothing on posix/nt. It calls pthread_atfork on pthread. In RTCollector.m3 give a child handler that stores FALSE in the three booleans that indicate the three threads have started. No prepare or parent handler I believe. Though I should check for global locks there. Probably the locks in ThreadPThread suffice? I wonder if we should only fork when the collector is turned off? In ThreadPThread.m3 provide three handlers: prepare: lock "everything", at least the 5 static locks. I might try walking all the threads and locking them too. parent: unlock everything child: unlock everything and reinitialize the globals Note that the atfork handlers run in many more programs than cvsup. They would be used also with fork + exec. fork + exec is probably relatively safe already. Perhaps a way to disable that -- no way in Posix, we could just provide a boolean to ourselves. Not sure I follow. The larger fix would be to provide a common/Thread.i3 function to call fork() *and* recreate all Modula-3 threads in the child process. That is what I sent out. That's tough to do portably! In the second case, you also then need to allow that the caller may or may not use that function, so you still need atfork handlers, but they have to be slightly different, as the diffs I sent show. I believe strongly this is the way to go. It is how one is a "good pthread citizen". Now, granted, I don't think most libraries are. Witness the caveat about fork() in the Darwin manpage and I think even the Posix spec. But pthread_atfork is meant to solve this. I don't suggest a full audit and add atfork handlers everywhere. But m3core we can at least do, and that should satisfy cvsup. Should check libm3 too. Really we only need to be concerned with internal threads to the run-time, right? It's up to apps to restart threads in the children as necessary. Plus, as long as each module takes reponsibility for itself, it shouldn't be so hard. That is, I don't think the case is all that strong for providing the "forkall" function that is in the diffs I sent. Yes, I think that is overkill. I'll do more testing and send out diffs "later". It could be as long as a week, I have a bunch of things to do. Or maybe just a day. :) I tested a form of this fix + cvsup + user threads and it appears that cvsup can do one small thing and thereby work either way, without knowledge as to the way. That is, it can shutdown the dispatcher "directly", instead of queuing to it. I'm not even sure the dispatcher thread is really needed. As I said, I don't believe the application can handle this all itself. Any library that uses pthreads is obligated to play into the general mechanism. Generally the internals are not exposed for the application to do it. Treating Modula-3 as a closed system in which nobody uses anything outside of or "below" m3core/libm3 I don't think is practical. "applications", arguably defines as "processes", are often composed of fairly disparate bodies of code that don't necessarily expose much to each other. For example, they might each create their own worker threads and not expose them. This is what pthread_atfork is for. Actually Tony, you gave me the lead here. :) ;-) There is a problem that m3posixthreads (user) and pthreads semantics diverage, and..I haven't been able to think of an way to fix that. Well, it is actually easy to make "forkall" the default and only behavior actually, on user and pthreads (but not NT). But I don't see a way to make "fork1" the behavior for user threads. You can associate a pid with each thread, and notice when the pid changes, but I don't know which one thread you would keep, and you wouldn't necessarily be able to register pthread_atfork handlers, unless maybe user threads were used only on platforms that actually have pthreads.. And still there's no good way to it on NT I expect. - Jay From: jay.krell at cornell.edu To: dragisha at m3w.org Date: Thu, 18 Mar 2010 10:09:38 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup Dragisha, Really? The server? With the -C flag and/or -f? Evidence is very very very very very good as to what is going on here. It is all related to "fork1" vs. "forkall". fork1 is the overwhelming usual behavior (Solaris has an option), and basically *any* library that uses pthreads is obligated to use pthread_atfork, be it a C library or the Modula-3 library, etc.. The application cannot do things, unless the library exposes its internal locks. The Posix spec for pthread_atfork explains the problem. Consider C code that links in Modula-3 code. C code is fairly free to use the fork + do work model. It isn't very common, but it does exist, and people have gone to extra length to keep it viable. bash and ccache I believe both use this model. Granted, if you had your own kernel thread implementation, it might have addressed this. I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions. The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork. I want to still try out the "bigger" change, but with the simpler lock/unlock. And test on Solaris and Darwin. I think for SuspendOthers to not work is not surprising. The threads can still hold a few locks even if suspended, I'm pretty sure. The point is to fork in a "controlled" fashion -- all locks held, and in the right order, so that both parent and child can release them. Taking "all" the locks in Prepare is more reliable. I do wonder if really "all" locks need to be taken. I only take the static ones declared in PThreadThread.c. - Jay > Subject: Re: [M3devel] cvsup > From: dragisha at m3w.org > To: jay.krell at cornell.edu > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > Yes, I can. I am building it regularly, from version to version, at > least few times a year. And it also worked with my ancient kernel thread > implementation. Was one of stress tests. > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 18 22:33:30 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 18 Mar 2010 17:33:30 -0400 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <1268828983.2906.0.camel@faramir.m3w.org>, , Message-ID: First off, AtFork probably does not belong in the Thread interface. That is part of the language definition and we shouldn't monkey with it. I guess I don't understand the specs of what you are proposing. I can't say that I can immediately imagine your diff. On 18 Mar 2010, at 16:55, Jay K wrote: > > Name should probably be something more like Thread.AtFork. > > > I disagree. > It should be PThreadAtFork, because it only does anything when > using pthreads. If you are on a non-pthread system (user threads or Win32), > I expect, I think, the function won't have any affect. > Granted, we could be using user threads on a system that has pthreads, > so there is ambiguity, there is ambiguity either way. > > > Does it mean, hey, thin wrapper over pthread_atfork, and I'll call it > even if using user threads? Or does it mean, only call it if > using pthreads? > > > > I wonder if we should only fork when the collector is turned off? > > > I don't quite follow. > You mean, anyone calling fork() must GC.Disable()? > Or LockHeap() > Or, take all the thread locks? > That last point is part of what I'm saying, since the "prepare" callback > would do that. > > > pthread_atfork lets you establish preconditions for fork() "distributed" > out around the code, including with access to internals. > Without having to "coordinate" with the caller of fork(). > > > > fork + exec is probably relatively safe already. > > > Right, but this change "unnecessarily" alters it. > The "prepare" callback would still run. > You can't discern fork + noexec vs. fork + exec > They start the same. > > > I think we are converging on agreement. > Let me do some more tuning (stylistic, diff reduction, not perf) and testing. > Diffs later, maybe tonight, maybe in a week, depending on life. > > > > Perhaps a way to disable that -- no way in Posix, we could just > > provide a boolean to ourselves. > > Not sure I follow. > > > I meant, something like, in m3core/libm3 where we have a fork+exec > sequence, set a boolean somewhere to tell all of our "prepare" handlers > not to waste their time doing anything. fork + exec need not > run the prepare handlers. > And I worry that all their lock taking might deadlock..but I suspect > that would indicate a bug. ? > > > > recreate the threads > > That's tough to do portably! > > > Easy with user threads -- automatically happens. > Easy with pthreads -- systems with fork(). > Hard/impossible on Win32. > > > > Really we only need to be concerned with internal threads > > to the run-time, right? It's up to apps to restart threads > > in the children as necessary. > > > I can agree with that. > It is kind of just a lazy app that says "hey, please recreate > all my threads, I didn't keep track of them, so I don't > know what to restart, I know you kept track of all of them, > so please do it for me". > > > You should be able to "imagine" what my diff looks like. > And maybe ok it without seeing it? > > > Anyway, I definitely want to see Darwin not hang first. > > > - Jay > > > > > > > > Subject: Re: [M3devel] cvsup > From: hosking at cs.purdue.edu > Date: Thu, 18 Mar 2010 13:16:03 -0400 > CC: dragisha at m3w.org; m3devel at elegosoft.com > To: jay.krell at cornell.edu > > On 18 Mar 2010, at 11:45, Jay K wrote: > > To repeat myself: > > > and basically *any* library that uses pthreads is obligated to use > > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > > This is the crux of the matter. > We are being "bad pthread citizens" by not using pthread_atfork. > Granted, we are probably in large company. > > > There is a smaller fix and a larger fix, but it seems very clear we > should take one of them. > > > The small fix is: > common/Thread.i3: add PThreadAtFork > > Name should probably be something more like Thread.AtFork. > > It does nothing on posix/nt. > It calls pthread_atfork on pthread. > > > In RTCollector.m3 give a child handler that stores FALSE in the > three booleans that indicate the three threads have started. > No prepare or parent handler I believe. > Though I should check for global locks there. > Probably the locks in ThreadPThread suffice? > > I wonder if we should only fork when the collector is turned off? > > In ThreadPThread.m3 provide three handlers: > prepare: lock "everything", at least the 5 static locks. > I might try walking all the threads and locking them too. > parent: unlock everything > child: unlock everything and reinitialize the globals > > > Note that the atfork handlers run in many more programs than cvsup. > They would be used also with fork + exec. > > fork + exec is probably relatively safe already. > > Perhaps a way to disable that -- no way in Posix, we could just > provide a boolean to ourselves. > > Not sure I follow. > > The larger fix would be to provide a common/Thread.i3 function > to call fork() *and* recreate all Modula-3 threads in the child process. > That is what I sent out. > > That's tough to do portably! > > In the second case, you also then need to allow that the caller > may or may not use that function, so you still need atfork handlers, > but they have to be slightly different, as the diffs I sent show. > > > I believe strongly this is the way to go. > It is how one is a "good pthread citizen". > Now, granted, I don't think most libraries are. > Witness the caveat about fork() in the Darwin manpage and > I think even the Posix spec. > But pthread_atfork is meant to solve this. > > > I don't suggest a full audit and add atfork handlers everywhere. > But m3core we can at least do, and that should satisfy cvsup. > Should check libm3 too. > > Really we only need to be concerned with internal threads to the run-time, right? It's up to apps to restart threads in the children as necessary. > > Plus, as long as each module takes reponsibility for itself, > it shouldn't be so hard. > That is, I don't think the case is all that strong for providing the "forkall" > function that is in the diffs I sent. > > Yes, I think that is overkill. > > I'll do more testing and send out diffs "later". > It could be as long as a week, I have a bunch of things to do. > Or maybe just a day. :) > > > I tested a form of this fix + cvsup + user threads and it appears > that cvsup can do one small thing and thereby work either way, > without knowledge as to the way. > > > That is, it can shutdown the dispatcher "directly", instead of queuing to it. > I'm not even sure the dispatcher thread is really needed. > > > As I said, I don't believe the application can handle this all itself. > Any library that uses pthreads is obligated to play into the general > mechanism. Generally the internals are not exposed for the application > to do it. > > Treating Modula-3 as a closed system in which nobody uses > anything outside of or "below" m3core/libm3 I don't think is practical. > > > "applications", arguably defines as "processes", are often composed > of fairly disparate bodies of code that don't necessarily expose much > to each other. For example, they might each create their own worker > threads and not expose them. > This is what pthread_atfork is for. > Actually Tony, you gave me the lead here. :) > > ;-) > > > > There is a problem that m3posixthreads (user) and pthreads semantics > diverage, and..I haven't been able to think of an way to fix that. > > > Well, it is actually easy to make "forkall" the default and only behavior actually, > on user and pthreads (but not NT). > > > But I don't see a way to make "fork1" the behavior for user threads. > You can associate a pid with each thread, and notice when the pid changes, > but I don't know which one thread you would keep, and you wouldn't > necessarily be able to register pthread_atfork handlers, unless maybe > user threads were used only on platforms that actually have pthreads.. > > > And still there's no good way to it on NT I expect. > > > - Jay > > From: jay.krell at cornell.edu > To: dragisha at m3w.org > Date: Thu, 18 Mar 2010 10:09:38 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > Dragisha, Really? The server? With the -C flag and/or -f? > > Evidence is very very very very very good as to what is going on here. > It is all related to "fork1" vs. "forkall". > fork1 is the overwhelming usual behavior (Solaris has an option), > and basically *any* library that uses pthreads is obligated to use > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > The application cannot do things, unless > the library exposes its internal locks. The Posix spec for > pthread_atfork explains the problem. > > Consider C code that links in Modula-3 code. > C code is fairly free to use the fork + do work model. It isn't very common, > but it does exist, and people have gone to extra length to keep it viable. > bash and ccache I believe both use this model. > > > Granted, if you had your own kernel thread implementation, it might > have addressed this. > > > I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions. > The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork. > I want to still try out the "bigger" change, but with the simpler lock/unlock. > And test on Solaris and Darwin. > > > I think for SuspendOthers to not work is not surprising. > The threads can still hold a few locks even if suspended, I'm pretty sure. > The point is to fork in a "controlled" fashion -- all locks held, and in > the right order, so that both parent and child can release them. > > > Taking "all" the locks in Prepare is more reliable. > > > I do wonder if really "all" locks need to be taken. > I only take the static ones declared in PThreadThread.c. > > - Jay > > > > Subject: Re: [M3devel] cvsup > > From: dragisha at m3w.org > > To: jay.krell at cornell.edu > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > Yes, I can. I am building it regularly, from version to version, at > > least few times a year. And it also worked with my ancient kernel thread > > implementation. Was one of stress tests. > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > -- > > Dragi?a Duri? > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 17:01:03 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements Message-ID: refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 18:21:13 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: Message-ID: diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: atfork2-cvsup.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: atfork2-m3core.txt URL: From jay.krell at cornell.edu Fri Mar 19 19:08:59 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 18:08:59 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , Message-ID: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 19 19:13:32 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 19 Mar 2010 14:13:32 -0400 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , Message-ID: Why not put it into RTProcess? Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > > could go in any of ThreadF, RTOS, RTProcess? > > > Or RTThread. Or almost anywhere. > > > Some typos in the diffs: > in a comment, ThreadF__ should be RTOS__ > function PThreadForkHandler named inconsistently, > should be PThreadAtForkHandler > > > Again, "PThread" appears in these names to indicate > their meaning is not portable. Their existance is, but not their meaning. > We have no way of saying "only call this function for pthreads", > instead as I understand, we provide a portable interface with > drastically varying semantics, such as "do something" vs. "do nothing". > > > Given that Init in ThreadPThread.m3 is private, we could probably > take the stackbase parameter from the caller instead. > > > The call to AtFork should probably follow InitWithStackbase, > in case it fails under low memory, the Die/assert more likely to "work". > > > I'm not completely happy making RTOS public. > So maybe ThreadF? > I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. > But for now they don't exist. > > > Maybe a new interface? ThreadPThread.i3, but is still > present on all platforms, so mostly portable can call it. > > > ThreadPThread.AtFork? > PThread.AtFork? That seems right. > > > - Jay > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 17:21:13 +0000 > Subject: Re: [M3devel] cvsup fix refinements > > diffs attached > > > For now I've removed the Get/SetNextForkWillExec. > I also think truly get/set is right, not inc/dec. > It was confusing enough to interpret and initialize > the value. And it is also not clear what the default > should be. The usual behavor is exec does follow fork. > The safer default if the handlers work ok is to assume > exec will not follow. It is a perf hint or a don't change > how things were hint? Hint to speed up fork+exec or hint > to avoid running the new code that might not work? > > > (For now I've removed the Get/SetNextForkWillExec.) > They'd have to be implemented three times, and > I had trouble defining them. > Obviously Win32/userthreads just return a > hardcoded true, but it is hard to explain > from the caller of Get's point of view. > > Maybe: > SetNextForkWillExec > GetNextForkWillExec > > and then: > PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? > > That is, > someone is who is calling fork and possibly exec, > can immediately tell what these functions mean. > > > But the implementer of an atfork handler has to > do quite a semantic translation I think. > I wrote it backwards the first time. That either > implies it is confusing, or I wasn't thinking. > > > If it really is a problem to run this stuff when exec > will follow, we should know quickly, as building cm3 > is an aggressive user of this code. > > > This version has worked repeatedly on Darwin. > I didn't test user threads but it is structured such > as to make that obviously work. > The cvsup changes are in pthreaad_atfork handlers, > so no change for userthreads. > > > I didn't test on others but confidence is quite high now. > > description of changes at least where they are hard to read: > > m3core: > Init split into Init and InitWithStackBase, > We have to provide the old stack base in the child fork handler > instead of estimating from the address of a local. I don't > know what stack is used, maybe the caller of fork? > i.e. we might have eaten a fair amount of stack and > there could be arbitrary references above the current stack. > > > WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner > > There is a resulting double lock in WeakRefFromRef, every time. > Could instead copy/paste more code around, i.e.: > EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > > > - Jay > > > > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 16:01:03 +0000 > Subject: [M3devel] cvsup fix refinements > > refinements: > > > new public functions in m3core: > > > TYPE PThreadForkHandler = PROCEDURE(); > > <* EXTERNAL RTOS__PThreadAtFork *> > PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; > > > (That's not a change yet.) > > > PROCEDURE SetNextForkWillExec(value: BOOLEAN); > > PROCEDURE GetNextForkWillExec(): BOOLEAN; > > These are only an optimization and should > perhaps be removed? They should be used > under LockHeap/UnlockHeap? which wasn't previously > public. This way, existing fork/exec paths > not affected, though maybe though might as well > ought to be? > > > could go in any of ThreadF, RTOS, RTProcess? > > > Should be called under RTOS.LockHeap? > Therefore I put in RTOS. > Also helps that RTOS is exported from *Thread.m3. > RTOS not previously public. > > > Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) > I implemented them that way: true incs, false decs. > > > Or don't bother with them at all? > > > Furthermore, in RTCollector, instead of claiming the threads > aren't started, if they were started, restart them. > This makes more sense for the weak cleaner to me, and > seems reasonable for the other two. > > > Diffs not yet available. > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 19:15:34 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 18:15:34 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , , Message-ID: RTProcess is fine. Unix is a little wierd because I don't think it should do anything if using userthreads. Unix implies a fairly thin wrapper? - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:13:32 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not put it into RTProcess?Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 19:47:46 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 18:47:46 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , , , , , , Message-ID: How about DoesForkExist, DoesForkInheritAllThreads? From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 18:15:34 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements RTProcess is fine. Unix is a little wierd because I don't think it should do anything if using userthreads. Unix implies a fairly thin wrapper? - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:13:32 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not put it into RTProcess?Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 19 19:47:00 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 19 Mar 2010 14:47:00 -0400 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , Message-ID: <62DD30EE-66EB-444C-84D0-0C1C0B56CD19@cs.purdue.edu> I think the atfork support should go into Unix.i3. It's not necessary for Windows, since Unix.fork is not available there. Also, we should make user and system threading behave the same way. So, for user threads we would destroy all the other threads in the child. This implies that we should have an m3core process fork mechanism. Anyone wanting to fork can then rely on the same behaviour. > Index: src/m3core.h > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/m3core.h,v > retrieving revision 1.8 > diff -u -r1.8 m3core.h > --- src/m3core.h 19 Mar 2010 17:07:58 -0000 1.8 > +++ src/m3core.h 19 Mar 2010 17:08:41 -0000 > @@ -404,6 +404,13 @@ > UINT32 __cdecl Uin__htonl(UINT32 x); > UINT16 __cdecl Uin__htons(UINT16 x); > > +typedef void (*PThreadForkHandler)(void); > + > +INTEGER > +RTOS__PThreadAtFork(PThreadForkHandler prep, > + PThreadForkHandler parent, > + PThreadForkHandler child); > + > #ifdef __cplusplus > } /* extern "C" */ > #endif > Index: src/runtime/common/RTCollector.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/runtime/common/RTCollector.m3,v > retrieving revision 1.88 > diff -u -r1.88 RTCollector.m3 > --- src/runtime/common/RTCollector.m3 15 Dec 2009 19:52:08 -0000 1.88 > +++ src/runtime/common/RTCollector.m3 19 Mar 2010 17:08:42 -0000 > @@ -2033,19 +2033,31 @@ > > VAR startedWeakCleaner := FALSE; > > -PROCEDURE WeakRefFromRef (r: REFANY; p: WeakRefCleanUpProc := NIL): WeakRef = > +PROCEDURE StartWeakCleaner () = > VAR > start := FALSE; > - result: WeakRef; > BEGIN > - <* ASSERT r # NIL *> > TRY > RTOS.LockHeap(); > - (* create a WeakCleaner thread the first time through *) > - IF p # NIL AND NOT startedWeakCleaner THEN > + IF NOT startedWeakCleaner THEN > start := TRUE; > startedWeakCleaner := TRUE; > END; > + FINALLY > + RTOS.UnlockHeap(); > + END; > + IF start THEN > + EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > + END; > + END StartWeakCleaner; > + > +PROCEDURE WeakRefFromRef (r: REFANY; p: WeakRefCleanUpProc := NIL): WeakRef = > + VAR > + result: WeakRef; > + BEGIN > + <* ASSERT r # NIL *> > + TRY > + RTOS.LockHeap(); > (* if necessary, expand weakTable *) > IF weakFree0 = -1 THEN ExpandWeakTable(); END; > IF p # NIL THEN > @@ -2075,9 +2087,8 @@ > FINALLY > RTOS.UnlockHeap(); > END; > - IF start THEN > - EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > - END; > + (* create a WeakCleaner thread the first time through *) > + StartWeakCleaner(); > RETURN result; > END WeakRefFromRef; I would leave WeakRefFromRef unchanged, and simply restart the thread in the child if startedWeakCleaner is TRUE: PROCEDURE AtForkChild() = BEGIN RTOS.LockHeap(); IF startedForeground THEN StartBackgroundCollection(); END; IF startedBackground THEN StartBackgroundCollection(); END; IF startedWeakCleaner THEN StarteWeakCleaner(); END: END AtForkChild; > > @@ -2771,8 +2782,24 @@ > > (*** INITIALIZATION ***) > > +PROCEDURE PThreadForkChild() = > + BEGIN > + IF startedForeground THEN > + StartBackgroundCollection(); > + END; > + IF startedBackground THEN > + StartBackgroundCollection(); > + END; > + IF startedWeakCleaner THEN > + StartWeakCleaner(); > + END; > + END PThreadForkChild; > + > PROCEDURE Init () = > + VAR r: INTEGER; > BEGIN > + r := RTOS.PThreadAtFork((*prepare*)NIL, (*parent*)NIL, PThreadForkChild); > + <* ASSERT r = 0 *> > IF RTParams.IsPresent("paranoidgc") THEN InstallSanityCheck(); END; > IF RTParams.IsPresent("nogc") THEN disableCount := 1; END; > IF RTParams.IsPresent("noincremental") THEN incremental := FALSE; END; > Index: src/runtime/common/RTOS.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/runtime/common/RTOS.i3,v > retrieving revision 1.10 > diff -u -r1.10 RTOS.i3 > --- src/runtime/common/RTOS.i3 8 Sep 2009 05:54:55 -0000 1.10 > +++ src/runtime/common/RTOS.i3 19 Mar 2010 17:08:42 -0000 > @@ -5,7 +5,7 @@ > (*| modified on Sun Feb 21 14:15:00 PST 1993 by jdd *) > (*| modified on Wed Jan 27 22:27:27 PST 1993 by mjordan *) > > -(* "RTOS" is a private interface that provides the low-level, > +(* "RTOS" is a semi-private interface that provides the low-level, > OS-specific memory allocation and shutdown routines. *) Leave this private! > > INTERFACE RTOS; > @@ -40,4 +40,13 @@ > (* Write the "n" bytes beginning at address "a" to the standard > error output file or console. *) > > +(* ThreadF__PThreadAtFork calls pthread_atfork > + * on pthreads platform, otherwise does nothing. > + * Return value is from pthread_atform, 0 for success, > + * always 0 with user threads and Win32 threads. > + *) > +TYPE PThreadForkHandler = PROCEDURE(); > +<* EXTERNAL RTOS__PThreadAtFork *> > +PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; > + Put the pthread_atfork wrapper into Unix.i3. > END RTOS. > Index: src/runtime/common/m3makefile > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/runtime/common/m3makefile,v > retrieving revision 1.27 > diff -u -r1.27 m3makefile > --- src/runtime/common/m3makefile 14 Dec 2009 08:09:48 -0000 1.27 > +++ src/runtime/common/m3makefile 19 Mar 2010 17:08:42 -0000 > @@ -58,7 +58,7 @@ > Interface ("RTSignal") > Interface ("RTStack") > Interface ("RTTypeSRC") > -interface ("RTOS") > +Interface ("RTOS") > > proc FileExists(a) is > return not stale (a, a) > Index: src/thread/POSIX/ThreadPosixC.c > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v > retrieving revision 1.32 > diff -u -r1.32 ThreadPosixC.c > --- src/thread/POSIX/ThreadPosixC.c 16 Dec 2009 12:21:36 -0000 1.32 > +++ src/thread/POSIX/ThreadPosixC.c 19 Mar 2010 17:08:42 -0000 > @@ -279,6 +279,14 @@ > #endif > } > > +INTEGER > +RTOS__PThreadAtFork(PThreadForkHandler prep, > + PThreadForkHandler parent, > + PThreadForkHandler child) > +{ > + return 0; > +} > + > #ifdef __cplusplus > } /* extern "C" */ > #endif > Index: src/thread/PTHREAD/ThreadPThread.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/PTHREAD/ThreadPThread.m3,v > retrieving revision 1.233 > diff -u -r1.233 ThreadPThread.m3 > --- src/thread/PTHREAD/ThreadPThread.m3 18 Mar 2010 11:12:10 -0000 1.233 > +++ src/thread/PTHREAD/ThreadPThread.m3 19 Mar 2010 17:08:42 -0000 > @@ -6,7 +6,7 @@ > Scheduler, SchedulerPosix, RTOS, RTHooks, ThreadPThread; > > IMPORT Cerrno, FloatMode, MutexRep, > - RTCollectorSRC, RTError, RTHeapRep, RTIO, RTParams, > + RTCollectorSRC, RTError, RTHeapRep, RTIO, RTOS, RTParams, > RTPerfTool, RTProcess, ThreadEvent, Time, > Unix, Utime, Word, Usched, > Uerror, Uexec; > @@ -1294,18 +1294,18 @@ > > (*-------------------------------------------------------- Initialization ---*) > > -PROCEDURE Init ()= > +PROCEDURE InitWithStackBase (stackbase: ADDRESS) = > VAR > self: T; > me: Activation; > BEGIN > - InitC(ADR(self)); > + InitC(stackbase); > > me := NEW(Activation, > mutex := pthread_mutex_new(), > cond := pthread_cond_new()); > InitActivations(me); > - me.stackbase := ADR(self); (* not quite accurate but hopefully ok *) > + me.stackbase := stackbase; > IF me.mutex = NIL OR me.cond = NIL THEN > Die(ThisLine(), "Thread initialization failed."); > END; > @@ -1322,8 +1322,48 @@ > IF RTParams.IsPresent("foregroundgc") THEN > RTCollectorSRC.StartForegroundCollection(); > END; > + END InitWithStackBase; > + > +PROCEDURE Init ()= > + VAR r: INTEGER; > + BEGIN > + r := RTOS.PThreadAtFork(PThreadAtForkPrepare, PThreadAtForkParent, PThreadAtForkChild); > + IF r # 0 THEN DieI(ThisLine(), r) END; > + InitWithStackBase(ADR(r)); (* not quite accurate but hopefully ok *) > END Init; > > +VAR locks := ARRAY [0..3] OF pthread_mutex_t{ > + activeMu, slotsMu, initMu, perfMu}; > + > +PROCEDURE PThreadAtForkPrepare() = > + VAR r: int; > + BEGIN > + joinMu.acquire(); > + LockHeap(); > + FOR i := FIRST(locks) TO LAST(locks) DO > + r := pthread_mutex_lock(locks[i]); > + IF r # 0 THEN DieI(ThisLine(), r) END; > + END; > + (* Walk activations and lock all threads? *) > + END PThreadAtForkPrepare; I think there may be deadlock issues here! Would it not be better simply to SuspendOthers() in prepare and ResumeOthers() in parent/child? You would still need to acquire slotsMu/initMu/perfMu before SuspendOthers, and release them after ResumeOthers(). > + > +PROCEDURE PThreadAtForkParent() = > + VAR r: int; > + BEGIN > + FOR i := LAST(locks) TO FIRST(locks) BY -1 DO > + r := pthread_mutex_unlock(locks[i]); > + IF r # 0 THEN DieI(ThisLine(), r) END; > + END; > + UnlockHeap(); > + joinMu.release(); > + END PThreadAtForkParent; > + > +PROCEDURE PThreadAtForkChild() = > + BEGIN > + PThreadAtForkParent(); > + InitWithStackBase(GetActivation().stackbase); > + END PThreadAtForkChild; > + > (*------------------------------------------------------------- collector ---*) > (* These procedures provide synchronization primitives for the allocator > and collector. *) > Index: src/thread/PTHREAD/ThreadPThreadC.c > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/PTHREAD/ThreadPThreadC.c,v > retrieving revision 1.123 > diff -u -r1.123 ThreadPThreadC.c > --- src/thread/PTHREAD/ThreadPThreadC.c 25 Feb 2010 08:31:36 -0000 1.123 > +++ src/thread/PTHREAD/ThreadPThreadC.c 19 Mar 2010 17:08:42 -0000 > @@ -414,6 +414,14 @@ > #endif > } > > +INTEGER > +RTOS__PThreadAtFork(PThreadForkHandler prep, > + PThreadForkHandler parent, > + PThreadForkHandler child) > +{ > + return pthread_atfork(prep, parent, child); > +} > + > #ifdef __cplusplus > } /* extern "C" */ > #endif > Index: src/thread/WIN32/ThreadWin32C.c > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32C.c,v > retrieving revision 1.64 > diff -u -r1.64 ThreadWin32C.c > --- src/thread/WIN32/ThreadWin32C.c 16 Jan 2010 12:44:56 -0000 1.64 > +++ src/thread/WIN32/ThreadWin32C.c 19 Mar 2010 17:08:42 -0000 > @@ -146,6 +146,15 @@ > p(bottom, __getReg(?)); /* bsp? bspstore? try numbers until we find it? */ > #endif > } > + > +/*-------------------------------------------------------------------------*/ > + > +INTEGER > +RTOS__PThreadAtFork(PThreadForkHandler prep, > + PThreadForkHandler parent, > + PThreadForkHandler child) > +{ > + return 0; > } > > /*-------------------------------------------------------------------------*/ On 19 Mar 2010, at 14:13, Tony Hosking wrote: > Why not put it into RTProcess? > Or into Unix? > I want to avoid confusion between Thread.Fork (fork a thread) and process fork. > > On 19 Mar 2010, at 14:08, Jay K wrote: > >> > could go in any of ThreadF, RTOS, RTProcess? >> >> >> Or RTThread. Or almost anywhere. >> >> >> Some typos in the diffs: >> in a comment, ThreadF__ should be RTOS__ >> function PThreadForkHandler named inconsistently, >> should be PThreadAtForkHandler >> >> >> Again, "PThread" appears in these names to indicate >> their meaning is not portable. Their existance is, but not their meaning. >> We have no way of saying "only call this function for pthreads", >> instead as I understand, we provide a portable interface with >> drastically varying semantics, such as "do something" vs. "do nothing". >> >> >> Given that Init in ThreadPThread.m3 is private, we could probably >> take the stackbase parameter from the caller instead. >> >> >> The call to AtFork should probably follow InitWithStackbase, >> in case it fails under low memory, the Die/assert more likely to "work". >> >> >> I'm not completely happy making RTOS public. >> So maybe ThreadF? >> I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. >> But for now they don't exist. >> >> >> Maybe a new interface? ThreadPThread.i3, but is still >> present on all platforms, so mostly portable can call it. >> >> >> ThreadPThread.AtFork? >> PThread.AtFork? That seems right. >> >> >> - Jay >> >> >> From: jay.krell at cornell.edu >> To: m3devel at elegosoft.com >> Date: Fri, 19 Mar 2010 17:21:13 +0000 >> Subject: Re: [M3devel] cvsup fix refinements >> >> diffs attached >> >> >> For now I've removed the Get/SetNextForkWillExec. >> I also think truly get/set is right, not inc/dec. >> It was confusing enough to interpret and initialize >> the value. And it is also not clear what the default >> should be. The usual behavor is exec does follow fork. >> The safer default if the handlers work ok is to assume >> exec will not follow. It is a perf hint or a don't change >> how things were hint? Hint to speed up fork+exec or hint >> to avoid running the new code that might not work? >> >> >> (For now I've removed the Get/SetNextForkWillExec.) >> They'd have to be implemented three times, and >> I had trouble defining them. >> Obviously Win32/userthreads just return a >> hardcoded true, but it is hard to explain >> from the caller of Get's point of view. >> >> Maybe: >> SetNextForkWillExec >> GetNextForkWillExec >> >> and then: >> PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? >> >> That is, >> someone is who is calling fork and possibly exec, >> can immediately tell what these functions mean. >> >> >> But the implementer of an atfork handler has to >> do quite a semantic translation I think. >> I wrote it backwards the first time. That either >> implies it is confusing, or I wasn't thinking. >> >> >> If it really is a problem to run this stuff when exec >> will follow, we should know quickly, as building cm3 >> is an aggressive user of this code. >> >> >> This version has worked repeatedly on Darwin. >> I didn't test user threads but it is structured such >> as to make that obviously work. >> The cvsup changes are in pthreaad_atfork handlers, >> so no change for userthreads. >> >> >> I didn't test on others but confidence is quite high now. >> >> description of changes at least where they are hard to read: >> >> m3core: >> Init split into Init and InitWithStackBase, >> We have to provide the old stack base in the child fork handler >> instead of estimating from the address of a local. I don't >> know what stack is used, maybe the caller of fork? >> i.e. we might have eaten a fair amount of stack and >> there could be arbitrary references above the current stack. >> >> >> WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner >> >> There is a resulting double lock in WeakRefFromRef, every time. >> Could instead copy/paste more code around, i.e.: >> EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); >> >> >> - Jay >> >> >> >> >> >> >> From: jay.krell at cornell.edu >> To: m3devel at elegosoft.com >> Date: Fri, 19 Mar 2010 16:01:03 +0000 >> Subject: [M3devel] cvsup fix refinements >> >> refinements: >> >> >> new public functions in m3core: >> >> >> TYPE PThreadForkHandler = PROCEDURE(); >> >> <* EXTERNAL RTOS__PThreadAtFork *> >> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; >> >> >> (That's not a change yet.) >> >> >> PROCEDURE SetNextForkWillExec(value: BOOLEAN); >> >> PROCEDURE GetNextForkWillExec(): BOOLEAN; >> >> These are only an optimization and should >> perhaps be removed? They should be used >> under LockHeap/UnlockHeap? which wasn't previously >> public. This way, existing fork/exec paths >> not affected, though maybe though might as well >> ought to be? >> >> >> could go in any of ThreadF, RTOS, RTProcess? >> >> >> Should be called under RTOS.LockHeap? >> Therefore I put in RTOS. >> Also helps that RTOS is exported from *Thread.m3. >> RTOS not previously public. >> >> >> Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) >> I implemented them that way: true incs, false decs. >> >> >> Or don't bother with them at all? >> >> >> Furthermore, in RTCollector, instead of claiming the threads >> aren't started, if they were started, restart them. >> This makes more sense for the weak cleaner to me, and >> seems reasonable for the other two. >> >> >> Diffs not yet available. >> >> >> - Jay > From hosking at cs.purdue.edu Fri Mar 19 19:49:14 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 19 Mar 2010 14:49:14 -0400 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , , Message-ID: <7654F263-E59A-4DC0-8225-0F5F3E5BB6C1@cs.purdue.edu> On 19 Mar 2010, at 14:15, Jay K wrote: > RTProcess is fine. > Unix is a little wierd because I don't think it should do anything if using userthreads. I think forking with user threads should have the same behaviour as with system threads: all threads except the forker will die. Another thing. What happens to the dead threads in the child when they get GC'd? Their finaliser will try to invoke CleanThread which might fail because the threads mutex/cond are in a bad state in the child. Is that possible? > Unix implies a fairly thin wrapper? > > - Jay > > > Subject: Re: [M3devel] cvsup fix refinements > From: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 14:13:32 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not put it into RTProcess? > Or into Unix? > I want to avoid confusion between Thread.Fork (fork a thread) and process fork. > > On 19 Mar 2010, at 14:08, Jay K wrote: > > > could go in any of ThreadF, RTOS, RTProcess? > > > Or RTThread. Or almost anywhere. > > > Some typos in the diffs: > in a comment, ThreadF__ should be RTOS__ > function PThreadForkHandler named inconsistently, > should be PThreadAtForkHandler > > > Again, "PThread" appears in these names to indicate > their meaning is not portable. Their existance is, but not their meaning. > We have no way of saying "only call this function for pthreads", > instead as I understand, we provide a portable interface with > drastically varying semantics, such as "do something" vs. "do nothing". > > > Given that Init in ThreadPThread.m3 is private, we could probably > take the stackbase parameter from the caller instead. > > > The call to AtFork should probably follow InitWithStackbase, > in case it fails under low memory, the Die/assert more likely to "work". > > > I'm not completely happy making RTOS public. > So maybe ThreadF? > I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. > But for now they don't exist. > > > Maybe a new interface? ThreadPThread.i3, but is still > present on all platforms, so mostly portable can call it. > > > ThreadPThread.AtFork? > PThread.AtFork? That seems right. > > > - Jay > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 17:21:13 +0000 > Subject: Re: [M3devel] cvsup fix refinements > > diffs attached > > > For now I've removed the Get/SetNextForkWillExec. > I also think truly get/set is right, not inc/dec. > It was confusing enough to interpret and initialize > the value. And it is also not clear what the default > should be. The usual behavor is exec does follow fork. > The safer default if the handlers work ok is to assume > exec will not follow. It is a perf hint or a don't change > how things were hint? Hint to speed up fork+exec or hint > to avoid running the new code that might not work? > > > (For now I've removed the Get/SetNextForkWillExec.) > They'd have to be implemented three times, and > I had trouble defining them. > Obviously Win32/userthreads just return a > hardcoded true, but it is hard to explain > from the caller of Get's point of view. > > Maybe: > SetNextForkWillExec > GetNextForkWillExec > > and then: > PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? > > That is, > someone is who is calling fork and possibly exec, > can immediately tell what these functions mean. > > > But the implementer of an atfork handler has to > do quite a semantic translation I think. > I wrote it backwards the first time. That either > implies it is confusing, or I wasn't thinking. > > > If it really is a problem to run this stuff when exec > will follow, we should know quickly, as building cm3 > is an aggressive user of this code. > > > This version has worked repeatedly on Darwin. > I didn't test user threads but it is structured such > as to make that obviously work. > The cvsup changes are in pthreaad_atfork handlers, > so no change for userthreads. > > > I didn't test on others but confidence is quite high now. > > description of changes at least where they are hard to read: > > m3core: > Init split into Init and InitWithStackBase, > We have to provide the old stack base in the child fork handler > instead of estimating from the address of a local. I don't > know what stack is used, maybe the caller of fork? > i.e. we might have eaten a fair amount of stack and > there could be arbitrary references above the current stack. > > > WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner > > There is a resulting double lock in WeakRefFromRef, every time. > Could instead copy/paste more code around, i.e.: > EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > > > - Jay > > > > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 16:01:03 +0000 > Subject: [M3devel] cvsup fix refinements > > refinements: > > > new public functions in m3core: > > > TYPE PThreadForkHandler = PROCEDURE(); > > <* EXTERNAL RTOS__PThreadAtFork *> > PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; > > > (That's not a change yet.) > > > PROCEDURE SetNextForkWillExec(value: BOOLEAN); > > PROCEDURE GetNextForkWillExec(): BOOLEAN; > > These are only an optimization and should > perhaps be removed? They should be used > under LockHeap/UnlockHeap? which wasn't previously > public. This way, existing fork/exec paths > not affected, though maybe though might as well > ought to be? > > > could go in any of ThreadF, RTOS, RTProcess? > > > Should be called under RTOS.LockHeap? > Therefore I put in RTOS. > Also helps that RTOS is exported from *Thread.m3. > RTOS not previously public. > > > Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) > I implemented them that way: true incs, false decs. > > > Or don't bother with them at all? > > > Furthermore, in RTCollector, instead of claiming the threads > aren't started, if they were started, restart them. > This makes more sense for the weak cleaner to me, and > seems reasonable for the other two. > > > Diffs not yet available. > > > - Jay > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 19 19:50:01 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 19 Mar 2010 14:50:01 -0400 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , , , , , , Message-ID: <83C24180-1290-4F1D-A777-D283475A543F@cs.purdue.edu> The way fork is defined on modern platforms I don't think the child should ever inherit all the threads. On 19 Mar 2010, at 14:47, Jay K wrote: > How about DoesForkExist, DoesForkInheritAllThreads? > > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 18:15:34 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup fix refinements > > RTProcess is fine. > Unix is a little wierd because I don't think it should do anything if using userthreads. > Unix implies a fairly thin wrapper? > > - Jay > > > Subject: Re: [M3devel] cvsup fix refinements > From: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 14:13:32 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not put it into RTProcess? > Or into Unix? > I want to avoid confusion between Thread.Fork (fork a thread) and process fork. > > On 19 Mar 2010, at 14:08, Jay K wrote: > > > could go in any of ThreadF, RTOS, RTProcess? > > > Or RTThread. Or almost anywhere. > > > Some typos in the diffs: > in a comment, ThreadF__ should be RTOS__ > function PThreadForkHandler named inconsistently, > should be PThreadAtForkHandler > > > Again, "PThread" appears in these names to indicate > their meaning is not portable. Their existance is, but not their meaning. > We have no way of saying "only call this function for pthreads", > instead as I understand, we provide a portable interface with > drastically varying semantics, such as "do something" vs. "do nothing". > > > Given that Init in ThreadPThread.m3 is private, we could probably > take the stackbase parameter from the caller instead. > > > The call to AtFork should probably follow InitWithStackbase, > in case it fails under low memory, the Die/assert more likely to "work". > > > I'm not completely happy making RTOS public. > So maybe ThreadF? > I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. > But for now they don't exist. > > > Maybe a new interface? ThreadPThread.i3, but is still > present on all platforms, so mostly portable can call it. > > > ThreadPThread.AtFork? > PThread.AtFork? That seems right. > > > - Jay > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 17:21:13 +0000 > Subject: Re: [M3devel] cvsup fix refinements > > diffs attached > > > For now I've removed the Get/SetNextForkWillExec. > I also think truly get/set is right, not inc/dec. > It was confusing enough to interpret and initialize > the value. And it is also not clear what the default > should be. The usual behavor is exec does follow fork. > The safer default if the handlers work ok is to assume > exec will not follow. It is a perf hint or a don't change > how things were hint? Hint to speed up fork+exec or hint > to avoid running the new code that might not work? > > > (For now I've removed the Get/SetNextForkWillExec.) > They'd have to be implemented three times, and > I had trouble defining them. > Obviously Win32/userthreads just return a > hardcoded true, but it is hard to explain > from the caller of Get's point of view. > > Maybe: > SetNextForkWillExec > GetNextForkWillExec > > and then: > PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? > > That is, > someone is who is calling fork and possibly exec, > can immediately tell what these functions mean. > > > But the implementer of an atfork handler has to > do quite a semantic translation I think. > I wrote it backwards the first time. That either > implies it is confusing, or I wasn't thinking. > > > If it really is a problem to run this stuff when exec > will follow, we should know quickly, as building cm3 > is an aggressive user of this code. > > > This version has worked repeatedly on Darwin. > I didn't test user threads but it is structured such > as to make that obviously work. > The cvsup changes are in pthreaad_atfork handlers, > so no change for userthreads. > > > I didn't test on others but confidence is quite high now. > > description of changes at least where they are hard to read: > > m3core: > Init split into Init and InitWithStackBase, > We have to provide the old stack base in the child fork handler > instead of estimating from the address of a local. I don't > know what stack is used, maybe the caller of fork? > i.e. we might have eaten a fair amount of stack and > there could be arbitrary references above the current stack. > > > WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner > > There is a resulting double lock in WeakRefFromRef, every time. > Could instead copy/paste more code around, i.e.: > EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > > > - Jay > > > > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 16:01:03 +0000 > Subject: [M3devel] cvsup fix refinements > > refinements: > > > new public functions in m3core: > > > TYPE PThreadForkHandler = PROCEDURE(); > > <* EXTERNAL RTOS__PThreadAtFork *> > PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; > > > (That's not a change yet.) > > > PROCEDURE SetNextForkWillExec(value: BOOLEAN); > > PROCEDURE GetNextForkWillExec(): BOOLEAN; > > These are only an optimization and should > perhaps be removed? They should be used > under LockHeap/UnlockHeap? which wasn't previously > public. This way, existing fork/exec paths > not affected, though maybe though might as well > ought to be? > > > could go in any of ThreadF, RTOS, RTProcess? > > > Should be called under RTOS.LockHeap? > Therefore I put in RTOS. > Also helps that RTOS is exported from *Thread.m3. > RTOS not previously public. > > > Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) > I implemented them that way: true incs, false decs. > > > Or don't bother with them at all? > > > Furthermore, in RTCollector, instead of claiming the threads > aren't started, if they were started, restart them. > This makes more sense for the weak cleaner to me, and > seems reasonable for the other two. > > > Diffs not yet available. > > > - Jay > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 20:56:04 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 19:56:04 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: <83C24180-1290-4F1D-A777-D283475A543F@cs.purdue.edu> References: , , , , , , , , , , , , <83C24180-1290-4F1D-A777-D283475A543F@cs.purdue.edu> Message-ID: Agreed. Details later. From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:50:01 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements The way fork is defined on modern platforms I don't think the child should ever inherit all the threads. On 19 Mar 2010, at 14:47, Jay K wrote:How about DoesForkExist, DoesForkInheritAllThreads? From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 18:15:34 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements RTProcess is fine. Unix is a little wierd because I don't think it should do anything if using userthreads. Unix implies a fairly thin wrapper? - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:13:32 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not put it into RTProcess?Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 19 21:17:34 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 19 Mar 2010 16:17:34 -0400 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , , , , , , , , , , , , <83C24180-1290-4F1D-A777-D283475A543F@cs.purdue.edu> Message-ID: <70C491E7-6699-4556-A92C-763B22CBD6DA@cs.purdue.edu> I meant for both user and system threads. On 19 Mar 2010, at 15:56, Jay K wrote: > Agreed. Details later. > > From: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 14:50:01 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup fix refinements > > The way fork is defined on modern platforms I don't think the child should ever inherit all the threads. > > On 19 Mar 2010, at 14:47, Jay K wrote: > > How about DoesForkExist, DoesForkInheritAllThreads? > > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 18:15:34 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup fix refinements > > RTProcess is fine. > Unix is a little wierd because I don't think it should do anything if using userthreads. > Unix implies a fairly thin wrapper? > > - Jay > > > Subject: Re: [M3devel] cvsup fix refinements > From: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 14:13:32 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not put it into RTProcess? > Or into Unix? > I want to avoid confusion between Thread.Fork (fork a thread) and process fork. > > On 19 Mar 2010, at 14:08, Jay K wrote: > > > could go in any of ThreadF, RTOS, RTProcess? > > > Or RTThread. Or almost anywhere. > > > Some typos in the diffs: > in a comment, ThreadF__ should be RTOS__ > function PThreadForkHandler named inconsistently, > should be PThreadAtForkHandler > > > Again, "PThread" appears in these names to indicate > their meaning is not portable. Their existance is, but not their meaning. > We have no way of saying "only call this function for pthreads", > instead as I understand, we provide a portable interface with > drastically varying semantics, such as "do something" vs. "do nothing". > > > Given that Init in ThreadPThread.m3 is private, we could probably > take the stackbase parameter from the caller instead. > > > The call to AtFork should probably follow InitWithStackbase, > in case it fails under low memory, the Die/assert more likely to "work". > > > I'm not completely happy making RTOS public. > So maybe ThreadF? > I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. > But for now they don't exist. > > > Maybe a new interface? ThreadPThread.i3, but is still > present on all platforms, so mostly portable can call it. > > > ThreadPThread.AtFork? > PThread.AtFork? That seems right. > > > - Jay > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 17:21:13 +0000 > Subject: Re: [M3devel] cvsup fix refinements > > diffs attached > > > For now I've removed the Get/SetNextForkWillExec. > I also think truly get/set is right, not inc/dec. > It was confusing enough to interpret and initialize > the value. And it is also not clear what the default > should be. The usual behavor is exec does follow fork. > The safer default if the handlers work ok is to assume > exec will not follow. It is a perf hint or a don't change > how things were hint? Hint to speed up fork+exec or hint > to avoid running the new code that might not work? > > > (For now I've removed the Get/SetNextForkWillExec.) > They'd have to be implemented three times, and > I had trouble defining them. > Obviously Win32/userthreads just return a > hardcoded true, but it is hard to explain > from the caller of Get's point of view. > > Maybe: > SetNextForkWillExec > GetNextForkWillExec > > and then: > PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? > > That is, > someone is who is calling fork and possibly exec, > can immediately tell what these functions mean. > > > But the implementer of an atfork handler has to > do quite a semantic translation I think. > I wrote it backwards the first time. That either > implies it is confusing, or I wasn't thinking. > > > If it really is a problem to run this stuff when exec > will follow, we should know quickly, as building cm3 > is an aggressive user of this code. > > > This version has worked repeatedly on Darwin. > I didn't test user threads but it is structured such > as to make that obviously work. > The cvsup changes are in pthreaad_atfork handlers, > so no change for userthreads. > > > I didn't test on others but confidence is quite high now. > > description of changes at least where they are hard to read: > > m3core: > Init split into Init and InitWithStackBase, > We have to provide the old stack base in the child fork handler > instead of estimating from the address of a local. I don't > know what stack is used, maybe the caller of fork? > i.e. we might have eaten a fair amount of stack and > there could be arbitrary references above the current stack. > > > WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner > > There is a resulting double lock in WeakRefFromRef, every time. > Could instead copy/paste more code around, i.e.: > EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > > > - Jay > > > > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 16:01:03 +0000 > Subject: [M3devel] cvsup fix refinements > > refinements: > > > new public functions in m3core: > > > TYPE PThreadForkHandler = PROCEDURE(); > > <* EXTERNAL RTOS__PThreadAtFork *> > PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; > > > (That's not a change yet.) > > > PROCEDURE SetNextForkWillExec(value: BOOLEAN); > > PROCEDURE GetNextForkWillExec(): BOOLEAN; > > These are only an optimization and should > perhaps be removed? They should be used > under LockHeap/UnlockHeap? which wasn't previously > public. This way, existing fork/exec paths > not affected, though maybe though might as well > ought to be? > > > could go in any of ThreadF, RTOS, RTProcess? > > > Should be called under RTOS.LockHeap? > Therefore I put in RTOS. > Also helps that RTOS is exported from *Thread.m3. > RTOS not previously public. > > > Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) > I implemented them that way: true incs, false decs. > > > Or don't bother with them at all? > > > Furthermore, in RTCollector, instead of claiming the threads > aren't started, if they were started, restart them. > This makes more sense for the weak cleaner to me, and > seems reasonable for the other two. > > > Diffs not yet available. > > > - Jay > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 23:49:15 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 22:49:15 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: <70C491E7-6699-4556-A92C-763B22CBD6DA@cs.purdue.edu> References: , , , , , , , , , , , , <83C24180-1290-4F1D-A777-D283475A543F@cs.purdue.edu> , <70C491E7-6699-4556-A92C-763B22CBD6DA@cs.purdue.edu> Message-ID: I was going to ask: Do you mean pthreads or Modula-3 threads? How you think it is or how you think it should be? Modula-3 user threads are inherited. That is what this function would say. At least they are currently. However, as you assert, I've also wondered many times to myself the past few days: Should Modula-3 userthreads and pthreads act the same way? Always? Optionally? What stopped me was I couldn't convince myself it is trivial to implement. I was thinking, like, you'd associate a pid with each thread. If getpid doesn't match thread.pid, the thread is from before a fork. I then worried that you'd build up all the data about threads "forever", across multiple forks. However that is probably false. As well, if you provide RTProcess.Fork() as the replacement for fork(), it affords more power or at least ease of implementation. I guess where you end is like: RTProcess.Fork() does nothing on Windows in fact is <* EXTERNAL *> and fails to link most likely You could have the convention of returning ENOSYS but I say no. Or it might be a NIL pointer, following the convention you know about, have used several times. just calls fork() on pthreads on user threads, described later RTProcess.AtFork() (maybe RegisterAtFork or RegisterForkCallbacks?) does nothing on Windows (but exists and links, usable by portable code) might be NIL on pthreads just calls pthread_atfork on user threads, described later And then RTProcess.Fork(), RTProcess.AtFork on user threads work together to provide basically the same meaning as fork() + pthread_fork() has on pthreads. Easy to imagine how to code it actually. AtFork maintains an array of function pointers. Fork() calls them and fork in the proscribed order. The array is inherited across processes (like pthread_atfork()). Only the calling thread survives, like fork(). I didn't realize it at first, but killing of all other threads is pretty easy. You just reinitialize like I did for pthreads. Then the next time the timer expires, it doesn't find them to be scheduled. I also strongly suspect that in libm3 where fork+exec is called, some of what it does -- in particular turning off the timer -- would be moved to RTProcess.Fork(). Or perhaps that is in ThreadPosix.m3's AtForkPrepare. Either way, the general idea is that fork is easier to use, be it fork + exec, or even fork + do work. You replace it with RTProcess.Fork(). As well however, fork + exec on pthreads continues to work. You don't have to call RTProcess.Fork(). However RTProcess.Fork() is needed to provide matching semantics in user threads if you don't exec after fork. Geez this is a mouthful. I think it is somewhat important to dwell on -- what semantics change, which external libraries just work and don't have to be wrapped. if you are doing fork + exec, you can keep doing the same, I think if you don't care about user threads, and want them all to be inherited, you can keep doing the same, I think No working code should have an altered meaning. cvsupd on pthreads doesn't count as working code, of course. In fact no fork + do work code on pthreads probably worked. My assertion that "we should use pthread_atfork and be good pthread citizens" remains true and respected by these changes. I'll code this up over the next few days/week. I'm not sure anything ends up in Unix, could all end up in RTProcessC.c or such. If RTProcess.i3 isn't public, it will be. Or we can introduce a new interface. This all sounds about right? Notice that, like my second set of diffs, existing pthreads fork + exec paths will be "significantly" affected -- taking all the locks in the prepare call. It should work. - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 16:17:34 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu I meant for both user and system threads. On 19 Mar 2010, at 15:56, Jay K wrote: Agreed. Details later. From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:50:01 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements The way fork is defined on modern platforms I don't think the child should ever inherit all the threads. On 19 Mar 2010, at 14:47, Jay K wrote: How about DoesForkExist, DoesForkInheritAllThreads? From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 18:15:34 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements RTProcess is fine. Unix is a little wierd because I don't think it should do anything if using userthreads. Unix implies a fairly thin wrapper? - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:13:32 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not put it into RTProcess? Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 23:58:08 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 22:58:08 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: <7654F263-E59A-4DC0-8225-0F5F3E5BB6C1@cs.purdue.edu> References: , , , , , , , <7654F263-E59A-4DC0-8225-0F5F3E5BB6C1@cs.purdue.edu> Message-ID: > What happens to the dead threads in the child when they get GC'd? > Their finaliser will try to invoke CleanThread which might fail because > the threads mutex/cond are in a bad state in the child. Is that possible? This is what pthread_atfork is for. I'm not sure you don't realize that, or are pointing out that I wasn't handling everything. You see the question mark in the ThreadPThread.AtForkPrepare in the diff I sent? That covers this..and I think indeed it needs work: I think for this reason Thread.AtForkPrepare should walk all the threads and take their locks, and then child will release them all, and we can go ahead and cleanup them up right away as well (in the child), we might otherwise lose all references to them, since they were in thread locals. I think AtForkPrepare needs to broadly speaking, acquire *all* locks. Not just some. The diffs I sent only take some. I will include this change in next set of diffs -- the ones that will also change user threads to not inherit threads after RTProcess.Fork(). I suspect this is a difficult task in some code bases. And I'm not sure I suggest clean this up throughout cm3. However we can at least address RTCollector.m3 and ThreadPThread.m3 and get cvsup working. And then we could be like Apple and the docs could just say "fork not followed by exec not guaranteed to work much". ? Also makes me wonder..we should provide wrapped up fork+exec, eh, we already do. It is in libm3. It should probably be in m3core, but I don't think that difference matters too much. - Jay From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:49:14 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements On 19 Mar 2010, at 14:15, Jay K wrote: RTProcess is fine. Unix is a little wierd because I don't think it should do anything if using userthreads. I think forking with user threads should have the same behaviour as with system threads: all threads except the forker will die. Another thing. What happens to the dead threads in the child when they get GC'd? Their finaliser will try to invoke CleanThread which might fail because the threads mutex/cond are in a bad state in the child. Is that possible? Unix implies a fairly thin wrapper? - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:13:32 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not put it into RTProcess? Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 20 04:17:07 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 20 Mar 2010 03:17:07 +0000 Subject: [M3devel] cvsup In-Reply-To: <1268916971.2586.3.camel@faramir.m3w.org> References: ,,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <1268828983.2906.0.camel@faramir.m3w.org>, , <1268916971.2586.3.camel@faramir.m3w.org> Message-ID: Dead as in was working, but no longer? Please elaborate if not working. What does it do? What platform? Can you please debug it? I been doing mostly RTIO.PutText/Flush anyway. I tried several "debuggers": gdb, m3gdb, dbx, couldn't get much out of any. - Jay > Subject: RE: [M3devel] cvsup > From: dragisha at m3w.org > To: jay.krell at cornell.edu > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > Date: Thu, 18 Mar 2010 13:56:11 +0100 > > Client, it's what I am using. And it's dead as we speak. > > Last worked before I pulled/built RC4 > > > On Thu, 2010-03-18 at 10:09 +0000, Jay K wrote: > > Dragisha, Really? The server? With the -C flag and/or -f? > > > > Evidence is very very very very very good as to what is going on here. > > It is all related to "fork1" vs. "forkall". > > fork1 is the overwhelming usual behavior (Solaris has an option), > > and basically *any* library that uses pthreads is obligated to use > > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > > > The application cannot do things, unless > > the library exposes its internal locks. The Posix spec for > > pthread_atfork explains the problem. > > > > Consider C code that links in Modula-3 code. > > C code is fairly free to use the fork + do work model. It isn't very > > common, > > but it does exist, and people have gone to extra length to keep it > > viable. > > bash and ccache I believe both use this model. > > > > > > Granted, if you had your own kernel thread implementation, it might > > have addressed this. > > > > > > I have a version now that has served over 1000 requests, similar to > > what I sent, but a bit reduced. The main change was I just called > > pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead > > of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just > > Linux. This version also doesn't expose the ForkProcessAndAllThreads > > functions. > > The client has to restart its threads, or in the cvsupd case, don't > > depend on the dispatcher thread to survive fork. > > I want to still try out the "bigger" change, but with the simpler > > lock/unlock. > > And test on Solaris and Darwin. > > > > > > I think for SuspendOthers to not work is not surprising. > > The threads can still hold a few locks even if suspended, I'm pretty > > sure. > > The point is to fork in a "controlled" fashion -- all locks held, and > > in > > the right order, so that both parent and child can release them. > > > > > > Taking "all" the locks in Prepare is more reliable. > > > > > > I do wonder if really "all" locks need to be taken. > > I only take the static ones declared in PThreadThread.c. > > > > - Jay > > > > > > > Subject: Re: [M3devel] cvsup > > > From: dragisha at m3w.org > > > To: jay.krell at cornell.edu > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > > > Yes, I can. I am building it regularly, from version to version, at > > > least few times a year. And it also worked with my ancient kernel > > thread > > > implementation. Was one of stress tests. > > > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > > > (Can anyone claim to have seen cvsup server work with kernel > > threads?) > > > -- > > > Dragi?a Duri? > > > > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 20 09:48:05 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 20 Mar 2010 08:48:05 +0000 Subject: [M3devel] pthread_atfork with user threads? Message-ID: Or should the userthreads version implement itself and require user to call RTProcess.Fork()? You know, we have this odd but ok/useful state: Even though we support userthreads, every system has pthreads. I still have to verify that all systems have pthread_atfork, but I assume so. Also funny thing btw, pthreads, userthreads, win32 threads, the code will be highly shared. They will call RegisterForkHandlers. The fork handlers will be same or similar. Specifically in ThreadPThread.c and ThreadPosix.c -- reinitialize. For ThreadPosix that should achieve "don't switch to any preexisting thread", which is equivalent to "kill all other threads". implied question: Call it AtFork or RegisterForkHandlers or ? RTProcessC.c typedef void (*ForkHandler)(void); #ifndef _WIN32 #include #include #include #endif /* NOTE: Even userthreads now depends * on availability of pthreads. * This can be fixed if need be. */ int RTProcess__RegisterForkHandlers( ForkHandler prepare, ForkHandler parent, ForkHandler child) { #ifdef _WIN32 return 0; #else while (1) { int i = pthread_atfork(prepare, parent, child); if (i != EAGAIN) return i; sleep(0); } #endif } -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 20 18:53:49 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 20 Mar 2010 17:53:49 +0000 Subject: [M3devel] atomics AMD64_DARWIN failures? Message-ID: new source -> compiling AtomicBoolean.m3 ../AMD64_DARWIN/AtomicBoolean.m3 => ../src/atomic/Atomic.mg: In function 'AtomicBoolean__Load': ../AMD64_DARWIN/AtomicBoolean.m3 => ../src/atomic/Atomic.mg:0: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicBoolean.mc new source -> compiling AtomicChar.m3 ../AMD64_DARWIN/AtomicChar.m3 => ../src/atomic/Atomic.mg: In function 'AtomicChar__Load': ../AMD64_DARWIN/AtomicChar.m3 => ../src/atomic/Atomic.mg:0: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicChar.mc new source -> compiling AtomicInteger.m3 ../AMD64_DARWIN/AtomicInteger.m3 => ../src/atomic/Atomic.mg: In function 'AtomicInteger__Store': ../AMD64_DARWIN/AtomicInteger.m3 => ../src/atomic/Atomic.mg:21: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicInteger.mc new source -> compiling AtomicLongint.m3 ../AMD64_DARWIN/AtomicLongint.m3 => ../src/atomic/Atomic.mg: In function 'AtomicLongint__Store': ../AMD64_DARWIN/AtomicLongint.m3 => ../src/atomic/Atomic.mg:21: internal compiler error: Bus error Please submit a full bug report, I'll look into it..first just rebuild from scratch. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 20 19:08:12 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 20 Mar 2010 18:08:12 +0000 Subject: [M3devel] atomics AMD64_DARWIN failures? In-Reply-To: References: Message-ID: Sorry, nevermind, new checkout, new build, works fine. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Sat, 20 Mar 2010 17:53:49 +0000 Subject: [M3devel] atomics AMD64_DARWIN failures? new source -> compiling AtomicBoolean.m3 ../AMD64_DARWIN/AtomicBoolean.m3 => ../src/atomic/Atomic.mg: In function 'AtomicBoolean__Load': ../AMD64_DARWIN/AtomicBoolean.m3 => ../src/atomic/Atomic.mg:0: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicBoolean.mc new source -> compiling AtomicChar.m3 ../AMD64_DARWIN/AtomicChar.m3 => ../src/atomic/Atomic.mg: In function 'AtomicChar__Load': ../AMD64_DARWIN/AtomicChar.m3 => ../src/atomic/Atomic.mg:0: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicChar.mc new source -> compiling AtomicInteger.m3 ../AMD64_DARWIN/AtomicInteger.m3 => ../src/atomic/Atomic.mg: In function 'AtomicInteger__Store': ../AMD64_DARWIN/AtomicInteger.m3 => ../src/atomic/Atomic.mg:21: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicInteger.mc new source -> compiling AtomicLongint.m3 ../AMD64_DARWIN/AtomicLongint.m3 => ../src/atomic/Atomic.mg: In function 'AtomicLongint__Store': ../AMD64_DARWIN/AtomicLongint.m3 => ../src/atomic/Atomic.mg:21: internal compiler error: Bus error Please submit a full bug report, I'll look into it..first just rebuild from scratch. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 20 21:52:52 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 20 Mar 2010 20:52:52 +0000 Subject: [M3devel] cvsup/fork fix Message-ID: attached is looking pretty good cvsup seems to work on Darwin, kernel threads or user threads I do see: 2010.03.20 13:47:08 PDT [62878]: -0 [0Kin+0Kout] ChannelMux protocol error: Expected EOF, didn't get it but it seems any combination does that. I could try again with no diffs and userthreads. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: atfork4-cvsup.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: atfork4-m3core.txt URL: From jay.krell at cornell.edu Sun Mar 21 02:34:56 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 21 Mar 2010 01:34:56 +0000 Subject: [M3devel] cvsup/fork fix Message-ID: > 2010.03.20 13:47:08 PDT [62878]: -0 [0Kin+0Kout] ChannelMux protocol error: Expected EOF, didn't get it I also see this *occasionally* with user threads and no changes, but very consistently with my changes. I'll probably look into it further. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: cvsup/fork fix Date: Sat, 20 Mar 2010 20:52:52 +0000 attached is looking pretty good cvsup seems to work on Darwin, kernel threads or user threads I do see: 2010.03.20 13:47:08 PDT [62878]: -0 [0Kin+0Kout] ChannelMux protocol error: Expected EOF, didn't get it but it seems any combination does that. I could try again with no diffs and userthreads. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 21 02:40:11 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 21 Mar 2010 01:40:11 +0000 Subject: [M3devel] recent commits Message-ID: commit mail is temporarily down (or is it just me?), so: http://www.modula3.com/cm3/ChangeLog-2010 2010-03-20 20:40 jkrell * m3-libs/m3core/src/runtime/common/RTProcessC.c: new file, empty to start 2010-03-20 07:59 jkrell * m3-libs/m3core/src/Csupport/Common/hand.c: implement set_gt(a,b) as set_lt(b,a) implement set_ge(a,b) as set_le(b,a) Really the frontend or backend should make this transform. 2010-03-20 07:56 jkrell * m3-libs/m3core/src/Csupport/Common/test_hand.c: add some tests for set_lt, le, gt, le; I find this code a bit hard to understand ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 21 08:19:08 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 21 Mar 2010 08:19:08 +0100 Subject: [M3devel] cvsup In-Reply-To: References: ,,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> ,, ,,<1268828983.2906.0.camel@faramir.m3w.org> , ,<1268916971.2586.3.camel@faramir.m3w.org> Message-ID: <1269155948.6829.5.camel@faramir.m3w.org> cvsup client was working for me all the time until latest build - RC4 pulled from local CVS (mirrored by cvsup) on Feb 21 2010. It also worked 5 yrs ago when I used it with my NPTL implementation. What little I understood glimpsing over this discussion, it affects cvsup server? Or is my situation typical? I'll try m3gdb it. Thanks On Sat, 2010-03-20 at 03:17 +0000, Jay K wrote: > Dead as in was working, but no longer? > Please elaborate if not working. What does it do? > What platform? > Can you please debug it? > I been doing mostly RTIO.PutText/Flush anyway. > I tried several "debuggers": gdb, m3gdb, dbx, couldn't get much out of > any. > > - Jay > > > Subject: RE: [M3devel] cvsup > > From: dragisha at m3w.org > > To: jay.krell at cornell.edu > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > Date: Thu, 18 Mar 2010 13:56:11 +0100 > > > > Client, it's what I am using. And it's dead as we speak. > > > > Last worked before I pulled/built RC4 > > > > > > On Thu, 2010-03-18 at 10:09 +0000, Jay K wrote: > > > Dragisha, Really? The server? With the -C flag and/or > -f? > > > > > > Evidence is very very very very very good as to what is going on > here. > > > It is all related to "fork1" vs. "forkall". > > > fork1 is the overwhelming usual behavior (Solaris has an option), > > > and basically *any* library that uses pthreads is obligated to use > > > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > > > > > The application cannot do things, unless > > > the library exposes its internal locks. The Posix spec for > > > pthread_atfork explains the problem. > > > > > > Consider C code that links in Modula-3 code. > > > C code is fairly free to use the fork + do work model. It isn't > very > > > common, > > > but it does exist, and people have gone to extra length to keep it > > > viable. > > > bash and ccache I believe both use this model. > > > > > > > > > Granted, if you had your own kernel thread implementation, it > might > > > have addressed this. > > > > > > > > > I have a version now that has served over 1000 requests, similar > to > > > what I sent, but a bit reduced. The main change was I just called > > > pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, > instead > > > of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just > > > Linux. This version also doesn't expose the > ForkProcessAndAllThreads > > > functions. > > > The client has to restart its threads, or in the cvsupd case, > don't > > > depend on the dispatcher thread to survive fork. > > > I want to still try out the "bigger" change, but with the simpler > > > lock/unlock. > > > And test on Solaris and Darwin. > > > > > > > > > I think for SuspendOthers to not work is not surprising. > > > The threads can still hold a few locks even if suspended, I'm > pretty > > > sure. > > > The point is to fork in a "controlled" fashion -- all locks held, > and > > > in > > > the right order, so that both parent and child can release them. > > > > > > > > > Taking "all" the locks in Prepare is more reliable. > > > > > > > > > I do wonder if really "all" locks need to be taken. > > > I only take the static ones declared in PThreadThread.c. > > > > > > - Jay > > > > > > > > > > Subject: Re: [M3devel] cvsup > > > > From: dragisha at m3w.org > > > > To: jay.krell at cornell.edu > > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > > > > > Yes, I can. I am building it regularly, from version to version, > at > > > > least few times a year. And it also worked with my ancient > kernel > > > thread > > > > implementation. Was one of stress tests. > > > > > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > > > > > (Can anyone claim to have seen cvsup server work with kernel > > > threads?) > > > > -- > > > > Dragi?a Duri? > > > > > > -- > > Dragi?a Duri? > > -- Dragi?a Duri? From jay.krell at cornell.edu Sun Mar 21 09:54:50 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 21 Mar 2010 08:54:50 +0000 Subject: [M3devel] cvsup In-Reply-To: <1269155948.6829.5.camel@faramir.m3w.org> References: ,,,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , , , <1268828983.2906.0.camel@faramir.m3w.org>, , , , <1268916971.2586.3.camel@faramir.m3w.org>, , <1269155948.6829.5.camel@faramir.m3w.org> Message-ID: Dragisha, it's still not clear to me if client is still working for not or not, on which platform, and if it doesn't work, what does it do? (Well, NPTL implies Linux.) Yes, mostly we are talking about server here. The server would always hang with kernel threads. But I'm going to look at client more due to this "EOF expected" warning I'm getting. > Or is my situation typical? What is your situation? We have very few reports of any kind, positive or negative. "Dead" could mean you stopped using it. But if it means it stopped working, please, what does it to? If it hangs, put in PutText+Flush calls to find out where it gets to before it hangs? See if it works with user threads (edit m3core/src/thread.quake)? - Jay > Subject: RE: [M3devel] cvsup > From: dragisha at m3w.org > To: jay.krell at cornell.edu > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > Date: Sun, 21 Mar 2010 08:19:08 +0100 > > cvsup client was working for me all the time until latest build - RC4 > pulled from local CVS (mirrored by cvsup) on Feb 21 2010. > > It also worked 5 yrs ago when I used it with my NPTL implementation. > > What little I understood glimpsing over this discussion, it affects > cvsup server? Or is my situation typical? > > I'll try m3gdb it. > > Thanks > > On Sat, 2010-03-20 at 03:17 +0000, Jay K wrote: > > Dead as in was working, but no longer? > > Please elaborate if not working. What does it do? > > What platform? > > Can you please debug it? > > I been doing mostly RTIO.PutText/Flush anyway. > > I tried several "debuggers": gdb, m3gdb, dbx, couldn't get much out of > > any. > > > > - Jay > > > > > Subject: RE: [M3devel] cvsup > > > From: dragisha at m3w.org > > > To: jay.krell at cornell.edu > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > Date: Thu, 18 Mar 2010 13:56:11 +0100 > > > > > > Client, it's what I am using. And it's dead as we speak. > > > > > > Last worked before I pulled/built RC4 > > > > > > > > > On Thu, 2010-03-18 at 10:09 +0000, Jay K wrote: > > > > Dragisha, Really? The server? With the -C flag and/or > > -f? > > > > > > > > Evidence is very very very very very good as to what is going on > > here. > > > > It is all related to "fork1" vs. "forkall". > > > > fork1 is the overwhelming usual behavior (Solaris has an option), > > > > and basically *any* library that uses pthreads is obligated to use > > > > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > > > > > > > The application cannot do things, unless > > > > the library exposes its internal locks. The Posix spec for > > > > pthread_atfork explains the problem. > > > > > > > > Consider C code that links in Modula-3 code. > > > > C code is fairly free to use the fork + do work model. It isn't > > very > > > > common, > > > > but it does exist, and people have gone to extra length to keep it > > > > viable. > > > > bash and ccache I believe both use this model. > > > > > > > > > > > > Granted, if you had your own kernel thread implementation, it > > might > > > > have addressed this. > > > > > > > > > > > > I have a version now that has served over 1000 requests, similar > > to > > > > what I sent, but a bit reduced. The main change was I just called > > > > pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, > > instead > > > > of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just > > > > Linux. This version also doesn't expose the > > ForkProcessAndAllThreads > > > > functions. > > > > The client has to restart its threads, or in the cvsupd case, > > don't > > > > depend on the dispatcher thread to survive fork. > > > > I want to still try out the "bigger" change, but with the simpler > > > > lock/unlock. > > > > And test on Solaris and Darwin. > > > > > > > > > > > > I think for SuspendOthers to not work is not surprising. > > > > The threads can still hold a few locks even if suspended, I'm > > pretty > > > > sure. > > > > The point is to fork in a "controlled" fashion -- all locks held, > > and > > > > in > > > > the right order, so that both parent and child can release them. > > > > > > > > > > > > Taking "all" the locks in Prepare is more reliable. > > > > > > > > > > > > I do wonder if really "all" locks need to be taken. > > > > I only take the static ones declared in PThreadThread.c. > > > > > > > > - Jay > > > > > > > > > > > > > Subject: Re: [M3devel] cvsup > > > > > From: dragisha at m3w.org > > > > > To: jay.krell at cornell.edu > > > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > > > > > > > Yes, I can. I am building it regularly, from version to version, > > at > > > > > least few times a year. And it also worked with my ancient > > kernel > > > > thread > > > > > implementation. Was one of stress tests. > > > > > > > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > > > > > > > (Can anyone claim to have seen cvsup server work with kernel > > > > threads?) > > > > > -- > > > > > Dragi?a Duri? > > > > > > > > -- > > > Dragi?a Duri? > > > > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 21 10:33:11 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 21 Mar 2010 10:33:11 +0100 Subject: [M3devel] cvsup In-Reply-To: References: ,,,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> ,,, ,,,<1268828983.2906.0.camel@faramir.m3w.org> ,, ,,<1268916971.2586.3.camel@faramir.m3w.org> , ,<1269155948.6829.5.camel@faramir.m3w.org> Message-ID: <1269163991.6829.15.camel@faramir.m3w.org> Few versions ago (May or July, when I last tried) cvsup started to behave ill - it just stops giving any output while memory usage is skyrocketing. With that version, I've synced "incrementally". When it locks, I kill it, then start over. With latest version, one from Feb 21, this does not help anymore. It just stops working (I've not observed if it leaks memory as former version). Half an hour ago I've reinstalled rpm's for one of this former versions I've had on my system. cvsup again sort-of-works. At least it finishes syncing after few restarts. This is what I get with -v: faramir:dragisha/pts/6: ~% cvsup -v CVSup client, non-GUI version Copyright 1996-2003 John D. Polstra Software version: 2009-08-03 16:02:04 Protocol version: 17.0 Operating system: LINUXLIBC6 It looks, for me, like client is culprit. Someone maybe can follow this even further back to find when and where. I don't think kernel threads are so guilty of this behaviour. I will build RC cvs head now. On Sun, 2010-03-21 at 08:54 +0000, Jay K wrote: > Dragisha, it's still not clear to me if client is still working for > not or not, > on which platform, and if it doesn't work, what does it do? > (Well, NPTL implies Linux.) > Yes, mostly we are talking about server here. > The server would always hang with kernel threads. > But I'm going to look at client more due to this "EOF expected" > warning I'm getting. > > > > Or is my situation typical? > > > What is your situation? > We have very few reports of any kind, positive or negative. > > > "Dead" could mean you stopped using it. > But if it means it stopped working, please, what does it to? > If it hangs, put in PutText+Flush calls to find out where > it gets to before it hangs? > See if it works with user threads (edit m3core/src/thread.quake)? > > > - Jay > > > Subject: RE: [M3devel] cvsup > > From: dragisha at m3w.org > > To: jay.krell at cornell.edu > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > Date: Sun, 21 Mar 2010 08:19:08 +0100 > > > > cvsup client was working for me all the time until latest build - > RC4 > > pulled from local CVS (mirrored by cvsup) on Feb 21 2010. > > > > It also worked 5 yrs ago when I used it with my NPTL implementation. > > > > What little I understood glimpsing over this discussion, it affects > > cvsup server? Or is my situation typical? > > > > I'll try m3gdb it. > > > > Thanks > > > > On Sat, 2010-03-20 at 03:17 +0000, Jay K wrote: > > > Dead as in was working, but no longer? > > > Please elaborate if not working. What does it do? > > > What platform? > > > Can you please debug it? > > > I been doing mostly RTIO.PutText/Flush anyway. > > > I tried several "debuggers": gdb, m3gdb, dbx, couldn't get much > out of > > > any. > > > > > > - Jay > > > > > > > Subject: RE: [M3devel] cvsup > > > > From: dragisha at m3w.org > > > > To: jay.krell at cornell.edu > > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > > Date: Thu, 18 Mar 2010 13:56:11 +0100 > > > > > > > > Client, it's what I am using. And it's dead as we speak. > > > > > > > > Last worked before I pulled/built RC4 > > > > > > > > > > > > On Thu, 2010-03-18 at 10:09 +0000, Jay K wrote: > > > > > Dragisha, Really? The server? With the -C flag and/or > > > -f? > > > > > > > > > > Evidence is very very very very very good as to what is going > on > > > here. > > > > > It is all related to "fork1" vs. "forkall". > > > > > fork1 is the overwhelming usual behavior (Solaris has an > option), > > > > > and basically *any* library that uses pthreads is obligated to > use > > > > > pthread_atfork, be it a C library or the Modula-3 library, > etc.. > > > > > > > > > > The application cannot do things, unless > > > > > the library exposes its internal locks. The Posix spec for > > > > > pthread_atfork explains the problem. > > > > > > > > > > Consider C code that links in Modula-3 code. > > > > > C code is fairly free to use the fork + do work model. It > isn't > > > very > > > > > common, > > > > > but it does exist, and people have gone to extra length to > keep it > > > > > viable. > > > > > bash and ccache I believe both use this model. > > > > > > > > > > > > > > > Granted, if you had your own kernel thread implementation, it > > > might > > > > > have addressed this. > > > > > > > > > > > > > > > I have a version now that has served over 1000 requests, > similar > > > to > > > > > what I sent, but a bit reduced. The main change was I just > called > > > > > pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, > > > instead > > > > > of using SuspendOthers. Haven't tested on Solaris/Darwin yet, > just > > > > > Linux. This version also doesn't expose the > > > ForkProcessAndAllThreads > > > > > functions. > > > > > The client has to restart its threads, or in the cvsupd case, > > > don't > > > > > depend on the dispatcher thread to survive fork. > > > > > I want to still try out the "bigger" change, but with the > simpler > > > > > lock/unlock. > > > > > And test on Solaris and Darwin. > > > > > > > > > > > > > > > I think for SuspendOthers to not work is not surprising. > > > > > The threads can still hold a few locks even if suspended, I'm > > > pretty > > > > > sure. > > > > > The point is to fork in a "controlled" fashion -- all locks > held, > > > and > > > > > in > > > > > the right order, so that both parent and child can release > them. > > > > > > > > > > > > > > > Taking "all" the locks in Prepare is more reliable. > > > > > > > > > > > > > > > I do wonder if really "all" locks need to be taken. > > > > > I only take the static ones declared in PThreadThread.c. > > > > > > > > > > - Jay > > > > > > > > > > > > > > > > Subject: Re: [M3devel] cvsup > > > > > > From: dragisha at m3w.org > > > > > > To: jay.krell at cornell.edu > > > > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > > > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > > > > > > > > > Yes, I can. I am building it regularly, from version to > version, > > > at > > > > > > least few times a year. And it also worked with my ancient > > > kernel > > > > > thread > > > > > > implementation. Was one of stress tests. > > > > > > > > > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > > > > > > > > > (Can anyone claim to have seen cvsup server work with > kernel > > > > > threads?) > > > > > > -- > > > > > > Dragi?a Duri? > > > > > > > > > > -- > > > > Dragi?a Duri? > > > > > > -- > > Dragi?a Duri? > > -- Dragi?a Duri? From dragisha at m3w.org Sun Mar 21 18:27:01 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 21 Mar 2010 18:27:01 +0100 Subject: [M3devel] cvsup (fix) state? Message-ID: <1269192421.6829.17.camel@faramir.m3w.org> I've just pulled latest cm3 RC and built up to cvsup... Started client, and it stalls - again - sometimes during process. Is this expected? Fix talk is still talk only? -- Dragi?a Duri? From Highjinks at gmx.com Tue Mar 23 21:02:32 2010 From: Highjinks at gmx.com (Chris) Date: Tue, 23 Mar 2010 21:02:32 +0100 (CET) Subject: [M3devel] Moving ahead with Modula3. Making my code safe. Message-ID: <20100323160650.74e96922.Highjinks@gmx.com> Alright, I'm finally comfortable enough with Modula3 to start doing some serious work with it. However, I'm now in the process of learning how to do things the Modula3 way. Or probably more correctly, writing things that are easily managed by other Modula3 developers. What is the "proper" Modula3 way of taking an unsafe interface to an external library and making it "safe" for the purposes of my fellow Modula3 developers? I know how I would it with Ada and GNAT GPL; but even though the languages are similiar, the environments are quite a bit different. The same thing goes for pretty much everything else. I really want to nail down my libraries and make them as bulletproof as possible. I'd like to make everything as exception free as possible. This means very few uses of the "RAISES" keyword. If a properly defined function returns a predefined error, I dont consider that exceptional. I typically consider any undefined behavior to be exceptional. Also, does Modula3 provide any facilities for Synchronous threads and/or State Machines/Automata, or do these all have to be coded manually? One last question, in regards to threads...how do I get access to Atomic ops? Typically I like to prefetch a dataset into the processors data cache and do most of my operations there. I can doubletime it on a multicore processor, assuming I dont have to use Mutexes and such. Any tips, pointers, or articles you might recommend? -- Chris From wagner at elegosoft.com Wed Mar 24 17:14:54 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Wed, 24 Mar 2010 17:14:54 +0100 Subject: [M3devel] Fwd: m3 Message-ID: <20100324171454.u9gw7n1bz4kcwcs8@mail.elegosoft.com> Any takers? All detail information missing so far :-( Olaf ----- Forwarded message from sunglee726 at gmail.com ----- Date: Wed, 24 Mar 2010 06:01:40 -0500 From: Sung Lee Reply-To: Sung Lee Subject: m3 To: m3-support at elego.de When I compile a modula-3 program and run it, I get an error saying the MSVCR90.dll was not found. What should I do? ----- End forwarded message ----- -------------- next part -------------- When I compile a modula-3 program and run it, I get an error saying the MSVCR90.dll was not found. What should I do? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 27 04:35:17 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 27 Mar 2010 03:35:17 +0000 Subject: [M3devel] interface UserThread? Message-ID: Should we offer, say, interface UserThread? One could say IMPORT UserThread AS Thread. It's pretty easy, on Posix. Except there isn't just Thread.T exposed by the thread library. There is e.g. Scheduler, SchedulerPosix. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Sat Mar 27 16:38:40 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Sat, 27 Mar 2010 16:38:40 +0100 Subject: [M3devel] interface UserThread? In-Reply-To: References: Message-ID: <20100327163840.81nebtotwkws0488@mail.elegosoft.com> Quoting Jay K : > Should we offer, say, interface UserThread? > One could say IMPORT UserThread AS Thread. > > It's pretty easy, on Posix. > Except there isn't just Thread.T exposed by the thread library. > There is e.g. Scheduler, SchedulerPosix. You'd like to mix those interfaces? Let's go one step back. What we would actually like to have is a simple switch for system and user threads, for example by just linking against another library. This isn't possible in the current implementation, because _everything_ would have to be recompiled, IIRC. I'm not sure about the reason anymore though. Why can't we have all thread-specific calls pass through one library interface? Does the compiler need to produce different code for some calls? Is it just too inefficient for some functions? If we cannot achieve something simpler that would not need changing the code, then I'd consider a new interface for user threads an option. I don't really like it, because Thread is a standard interface of the language, and we should be very careful to change anything there. At least we should agree that it's worth the efforts. Any opinions? Olaf From wagner at elegosoft.com Sat Mar 27 17:05:53 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Sat, 27 Mar 2010 17:05:53 +0100 Subject: [M3devel] Moving ahead with Modula3. Making my code safe. In-Reply-To: <20100323160650.74e96922.Highjinks@gmx.com> References: <20100323160650.74e96922.Highjinks@gmx.com> Message-ID: <20100327170553.wn2dv2t3448880cs@mail.elegosoft.com> Quoting Chris : > Alright, I'm finally comfortable enough with Modula3 to start doing > some serious work with it. That sounds promising :-) > However, I'm now in the process of learning how to do things the > Modula3 way. Or probably more correctly, writing things that are > easily managed by other Modula3 developers. > > What is the "proper" Modula3 way of taking an unsafe interface to an > external library and making it "safe" for the purposes of my fellow > Modula3 developers? I know how I would it with Ada and GNAT GPL; > but even though the languages are similiar, the environments are > quite a bit different. I don't think you can really make external C and C++ code safe in the sense that Modula-3 code is safe. What you can do is to make sure that the types are mapped correctly, there are no memory leaks and memory overwrites, that all parameters actually are in the expected range, and the returned values are interpreted correctly in the enclosing M3 code. Interfaces to C or C++ are never safe. You can do your best to make the module that uses them safe, i.e. there should be no undetected runtime error or unnoticed side-effect by calling methods of your module. > The same thing goes for pretty much everything else. I really want > to nail down my libraries and make them as bulletproof as possible. Yes, that's the general idea :-) > I'd like to make everything as exception free as possible. This > means very few uses of the "RAISES" keyword. If a properly defined > function returns a predefined error, I dont consider that > exceptional. I typically consider any undefined behavior to be > exceptional. This depends. Exceptions, as the name suggests, should be used for exceptional situations that occur rather infrequently. They may also impose some performance penalty depending on their implementation. The usual, error-free code path should not involve any exceptions. > Also, does Modula3 provide any facilities for Synchronous threads > and/or State Machines/Automata, or do these all have to be coded > manually? Offhand, I don't know of any modules for coroutines or finite state machines. Regarding lexical analysis, you will surely find some FSM-based scanner generators in the compiler tools (caltech-parser). > One last question, in regards to threads...how do I get access to > Atomic ops? Typically I like to prefetch a dataset into the > processors data cache and do most of my operations there. I can > doubletime it on a multicore processor, assuming I dont have to use > Mutexes and such. Atomic operations on processor level are or course very platform specific, and are not exposed in the standard libraries. The standard way to synchronize resource access and make certain operations atomic from an external view is to use mutexes, which are defined in the Thread interface. If this is too high-level or inefficient for certain purposes, Antony Hosking has recently added some generic atomics support in m3core/src/atomic/Atomic.ig. I'm pretty sure it is only in the current head though, and don't know if it's really already implemented on all target platforms. > Any tips, pointers, or articles you might recommend? The language definition as well as a tutorial and a bibliography are online, though I'm afraid that several of the links to external resources are outdated. You surely have browsed through the documentation section of http://www.opencm3.net/? Hope this helps, Olaf From wagner at elegosoft.com Sat Mar 27 17:28:52 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Sat, 27 Mar 2010 17:28:52 +0100 Subject: [M3devel] m3 In-Reply-To: <91819bfb1003240401h6ee7d8fdr70cd68c174cc5d9@mail.gmail.com> References: <91819bfb1003240401h6ee7d8fdr70cd68c174cc5d9@mail.gmail.com> Message-ID: <20100327172852.z6f981if40ww4sok@mail.elegosoft.com> Quoting Sung Lee : > When I compile a modula-3 program and run it, I get an error saying the > MSVCR90.dll was not found. What should I do? Sorry for the delay, I've been very busy again during the week. I forwarded your help request to the m3devel mailing list, but nobody there seems to have been eager to act on it, too. This could be due to the little information you provide though. What exact version of Modula-3 are you using? What commands did you issue, and what C compiler/linker from Microsoft have you installed? It may simply be that the dll in question is not in your path, but I'm not very familiar with Windows systems, so this is just a guess. If you provide more details, others may be able to help. You may also want to create a trac ticket: https://projects.elego.de/cm3/newticket Be sure to fill out all the fields with as much information as possible. Best regards, Olaf From wagner at elegosoft.com Sat Mar 27 18:46:59 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Sat, 27 Mar 2010 18:46:59 +0100 Subject: [M3devel] cvsup/fork fix In-Reply-To: References: Message-ID: <20100327184659.4ipyw1tdicsko0sk@mail.elegosoft.com> Quoting Jay K : > attached is looking pretty good > cvsup seems to work on Darwin, kernel threads or user threads > > I do see: > 2010.03.20 13:47:08 PDT [62878]: -0 [0Kin+0Kout] ChannelMux protocol > error: Expected EOF, didn't get it > > but it seems any combination does that. > I could try again with no diffs and userthreads. What's the status of the fork/threads fixes needed to make CVSup work again? Any chance to get something into the release branch in order to go ahead with RC5? There was also an issue reported with the client. I tried to run it to update the CM3 repository, and it stalls immediately: % /usr/local/cm3/bin/cvsup -g -L 2 -P m -l LOCK_cvsup cvsupfile.cm3 Parsing supfile "cvsupfile.cm3" Connecting to birch.elego.de Connected to birch.elego.de Server software version: release_DCVS_1_0_3_rc8_at_old_elego_de Negotiating file attribute support Exchanging collection information Establishing multiplexed-mode data connection If I run it in the FreeBSD gdb though, it doesn't hang and works as expected (apart from the unnecessary SetAttrs operations). This is with the latest sources from the release branch. It's not deterministic though; sometimes it gets on with its work. I'll try again to catch the deadlock in the debugger. Any other ideas? Olaf From jay.krell at cornell.edu Sun Mar 28 03:26:54 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 28 Mar 2010 01:26:54 +0000 Subject: [M3devel] cvsup/fork Message-ID: The "unexpected EOF" warnings in cvsup worry me, but I'm not sure I have a chance of figuring them out particularly soon. I fear I'll have to read the code a fair amount and look for race conditions, given the intermittent behavior. The changes in m3core which make it work better are correct seemingly independently. I'm inclined to just go with them, at least the pthread part. The EOF warning shows up in the current version even, with user threads, about 1 in 10 times. I'm not seeing m3commit mails. Just me? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 28 08:51:27 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 28 Mar 2010 06:51:27 +0000 Subject: [M3devel] cvsup/SigHandler.m3? Message-ID: This whole file SigHandler.m3 exists so that Modula-3 signal handlers are run in a special thread, free to use any Modula-3 facility. Is it really needed? Is it needed when using pthreads? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From mand at elegosoft.com Sun Mar 28 09:31:41 2010 From: mand at elegosoft.com (mand at elegosoft.com) Date: Sun, 28 Mar 2010 09:31:41 +0200 Subject: [M3devel] test mail Message-ID: <20100328093141.hbtyulj8oo008skg@mail.elego.de> checking for relay to non-elego domains. From jay.krell at cornell.edu Sun Mar 28 10:44:39 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 28 Mar 2010 08:44:39 +0000 Subject: [M3devel] cvsup/fork Message-ID: Mostly nevermind. I'm not seeing the EOF warnings on Linux/x86. Maybe they were Darwin only. I'll look into that "later", but based on Linux/x86 I'll proceed with the fixes. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: cvsup/fork Date: Sun, 28 Mar 2010 01:26:54 +0000 The "unexpected EOF" warnings in cvsup worry me, but I'm not sure I have a chance of figuring them out particularly soon. I fear I'll have to read the code a fair amount and look for race conditions, given the intermittent behavior. The changes in m3core which make it work better are correct seemingly independently. I'm inclined to just go with them, at least the pthread part. The EOF warning shows up in the current version even, with user threads, about 1 in 10 times. I'm not seeing m3commit mails. Just me? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 28 11:58:04 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 28 Mar 2010 09:58:04 +0000 Subject: [M3devel] userthreads vs. pthreads performance? Message-ID: I understand that userthreads don't scale across multiple processors. But testing out cvsup lately, things are noticably much much faster with user threads. At least cvsup. I haven't watched the compiler and such. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 28 12:15:57 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 28 Mar 2010 12:15:57 +0200 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: Message-ID: <1269771357.3671.7.camel@faramir.m3w.org> CVSup w/ user threads is not faster because user threads are faster - it is faster because it's buggy behaviour manifests only with kernel threads. IMO. Algorithm-wise - at least on Linux NPTL is O(1) decision making time, ie switching threads. User threads are lots of linear list walking. Figure how efficient it is. Process management "science" has gone long way since "founding fathers" did user threads thingy. d On Sun, 2010-03-28 at 09:58 +0000, Jay K wrote: > I understand that userthreads don't scale across multiple processors. > But testing out cvsup lately, things are noticably much much faster > with user threads. At least cvsup. I haven't watched the compiler and > such. > > > - Jay > > > > > -- Dragi?a Duri? From mika at async.async.caltech.edu Sun Mar 28 18:24:24 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Sun, 28 Mar 2010 09:24:24 -0700 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269771357.3671.7.camel@faramir.m3w.org> References: <1269771357.3671.7.camel@faramir.m3w.org> Message-ID: <20100328162424.670F91A20B7@async.async.caltech.edu> In my tests, programs that acquire a lot of locks (even uncontended ones, I think) run much much faster with user threads than with kernel threads. Or at least they run much (several times) faster compiled with an ancient PM3 than with a recent CM3, and I'm pretty sure the threading is the main difference. (Given that the effect only occurs for programs that acquire a lot of locks.) Mika =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: >CVSup w/ user threads is not faster because user threads are faster - it >is faster because it's buggy behaviour manifests only with kernel >threads. IMO. > >Algorithm-wise - at least on Linux NPTL is O(1) decision making time, ie >switching threads. User threads are lots of linear list walking. Figure >how efficient it is. Process management "science" has gone long way >since "founding fathers" did user threads thingy. > >d > >On Sun, 2010-03-28 at 09:58 +0000, Jay K wrote: >> I understand that userthreads don't scale across multiple processors. >> But testing out cvsup lately, things are noticably much much faster >> with user threads. At least cvsup. I haven't watched the compiler and >> such. >> >> >> - Jay >> >> >> >> >> >-- >Dragi??a Duri?? From hosking at cs.purdue.edu Sun Mar 28 18:55:36 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 28 Mar 2010 12:55:36 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <20100328162424.670F91A20B7@async.async.caltech.edu> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> Message-ID: <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> Yes, we need to implement thin locks and biased locks. These have minimal overhead for locks held mostly by just one thread. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 28 Mar 2010, at 12:24, Mika Nystrom wrote: > > In my tests, programs that acquire a lot of locks (even uncontended ones, > I think) run much much faster with user threads than with kernel threads. > Or at least they run much (several times) faster compiled with an ancient > PM3 than with a recent CM3, and I'm pretty sure the threading is the > main difference. (Given that the effect only occurs for programs that > acquire a lot of locks.) > > Mika > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: >> CVSup w/ user threads is not faster because user threads are faster - it >> is faster because it's buggy behaviour manifests only with kernel >> threads. IMO. >> >> Algorithm-wise - at least on Linux NPTL is O(1) decision making time, ie >> switching threads. User threads are lots of linear list walking. Figure >> how efficient it is. Process management "science" has gone long way >> since "founding fathers" did user threads thingy. >> >> d >> >> On Sun, 2010-03-28 at 09:58 +0000, Jay K wrote: >>> I understand that userthreads don't scale across multiple processors. >>> But testing out cvsup lately, things are noticably much much faster >>> with user threads. At least cvsup. I haven't watched the compiler and >>> such. >>> >>> >>> - Jay >>> >>> >>> >>> >>> >> -- >> Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 28 20:32:22 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 28 Mar 2010 20:32:22 +0200 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> Message-ID: <1269801143.3671.17.camel@faramir.m3w.org> Fast glance shows there is no overhead in code? Just passthrough of MUTEX field to pthread_mutex_lock()? And inefficiency comes from there? On Sun, 2010-03-28 at 12:55 -0400, Tony Hosking wrote: > Yes, we need to implement thin locks and biased locks. These have > minimal overhead for locks held mostly by just one thread. -- Dragi?a Duri? From mika at async.async.caltech.edu Sun Mar 28 20:57:40 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Sun, 28 Mar 2010 11:57:40 -0700 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269801143.3671.17.camel@faramir.m3w.org> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> <1269801143.3671.17.camel@faramir.m3w.org> Message-ID: <20100328185740.38AFA1A20B9@async.async.caltech.edu> Yep, sounds right. I was profiling some other thread-using code that slowed down enormously because of pthreads and it turned out the program was spending ~95% of its time in accessing the thread locals via one of the pthread_ functions. (The overhead of entering the kernel.) Mika =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: >Fast glance shows there is no overhead in code? Just passthrough of >MUTEX field to pthread_mutex_lock()? And inefficiency comes from there? > >On Sun, 2010-03-28 at 12:55 -0400, Tony Hosking wrote: >> Yes, we need to implement thin locks and biased locks. These have >> minimal overhead for locks held mostly by just one thread. >-- >Dragi??a Duri?? From dragisha at m3w.org Sun Mar 28 21:02:04 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 28 Mar 2010 21:02:04 +0200 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <20100328185740.38AFA1A20B9@async.async.caltech.edu> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> <1269801143.3671.17.camel@faramir.m3w.org> <20100328185740.38AFA1A20B9@async.async.caltech.edu> Message-ID: <1269802924.3671.18.camel@faramir.m3w.org> Which platform? On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > Yep, sounds right. > > I was profiling some other thread-using code that slowed down > enormously > because of pthreads and it turned out the program was spending ~95% > of its time in accessing the thread locals via one of the pthread_ > functions. > (The overhead of entering the kernel.) -- Dragi?a Duri? From hosking at cs.purdue.edu Sun Mar 28 21:08:12 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 28 Mar 2010 15:08:12 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269801143.3671.17.camel@faramir.m3w.org> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> <1269801143.3671.17.camel@faramir.m3w.org> Message-ID: <13B06EBC-1349-4973-BD2B-62C061509471@cs.purdue.edu> Yep. Call. Atomic ops. All expensive. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 28 Mar 2010, at 14:32, Dragi?a Duri? wrote: > Fast glance shows there is no overhead in code? Just passthrough of > MUTEX field to pthread_mutex_lock()? And inefficiency comes from there? > > On Sun, 2010-03-28 at 12:55 -0400, Tony Hosking wrote: >> Yes, we need to implement thin locks and biased locks. These have >> minimal overhead for locks held mostly by just one thread. > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.async.caltech.edu Sun Mar 28 21:11:16 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Sun, 28 Mar 2010 12:11:16 -0700 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269802924.3671.18.camel@faramir.m3w.org> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> <1269801143.3671.17.camel@faramir.m3w.org> <20100328185740.38AFA1A20B9@async.async.caltech.edu> <1269802924.3671.18.camel@faramir.m3w.org> Message-ID: <20100328191116.CD9C71A20BB@async.async.caltech.edu> Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: >Which platform? > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: >> Yep, sounds right. >> >> I was profiling some other thread-using code that slowed down >> enormously >> because of pthreads and it turned out the program was spending ~95% >> of its time in accessing the thread locals via one of the pthread_ >> functions. >> (The overhead of entering the kernel.) >-- >Dragi??a Duri?? From dragisha at m3w.org Sun Mar 28 21:14:57 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 28 Mar 2010 21:14:57 +0200 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <20100328191116.CD9C71A20BB@async.async.caltech.edu> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> <1269801143.3671.17.camel@faramir.m3w.org> <20100328185740.38AFA1A20B9@async.async.caltech.edu> <1269802924.3671.18.camel@faramir.m3w.org> <20100328191116.CD9C71A20BB@async.async.caltech.edu> Message-ID: <1269803697.3671.19.camel@faramir.m3w.org> I remember reading (long time ago) about how these (FUTEXes) are efficient in LINUX... Can I have your test code to try? On Sun, 2010-03-28 at 12:11 -0700, Mika Nystrom wrote: > Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > >Which platform? > > > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > >> Yep, sounds right. > >> > >> I was profiling some other thread-using code that slowed down > >> enormously > >> because of pthreads and it turned out the program was spending ~95% > >> of its time in accessing the thread locals via one of the pthread_ > >> functions. > >> (The overhead of entering the kernel.) > >-- > >Dragi?a Duri? -- Dragi?a Duri? From mika at async.async.caltech.edu Sun Mar 28 22:33:53 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Sun, 28 Mar 2010 13:33:53 -0700 Subject: [M3devel] Extended Static Checker Message-ID: <20100328203353.8507A1A20BB@async.async.caltech.edu> Hello m3devel, I am curious if the source code for the Modula-3 Extended Static Checker is available anywhere? Or a Linux binary might work for my purposes... I seem to remember they did distribute it at some point... Mika From jay.krell at cornell.edu Sun Mar 28 22:46:01 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 28 Mar 2010 20:46:01 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269803697.3671.19.camel@faramir.m3w.org> References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> Message-ID: O(1) scheduling is not a new idea. Just look at NT and probably Solaris and probably all the other non-free systems (AIX, Irix, HP-UX, Tru64, VMS, etc.) Getting thread locals should not require a kernel call. It doesn't on NT. We can optimize this somewhat on most systems with __thread. I had that in briefly. Entering an uncontended pthread mutex should not be expensive -- at least no kernel call, but granted a call and atomic op. Two calls because of the C layer. But user threads pay for a call too of course. Maybe I should profile some of this.. - Jay > From: dragisha at m3w.org > To: mika at async.async.caltech.edu > Date: Sun, 28 Mar 2010 21:14:57 +0200 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > I remember reading (long time ago) about how these (FUTEXes) are > efficient in LINUX... Can I have your test code to try? > > On Sun, 2010-03-28 at 12:11 -0700, Mika Nystrom wrote: > > Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... > > > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > > >Which platform? > > > > > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > > >> Yep, sounds right. > > >> > > >> I was profiling some other thread-using code that slowed down > > >> enormously > > >> because of pthreads and it turned out the program was spending ~95% > > >> of its time in accessing the thread locals via one of the pthread_ > > >> functions. > > >> (The overhead of entering the kernel.) > > >-- > > >Dragi?a Duri? > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Highjinks at gmx.com Sun Mar 28 23:01:31 2010 From: Highjinks at gmx.com (Chris) Date: Sun, 28 Mar 2010 23:01:31 +0200 (CEST) Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: Message-ID: <20100328170619.5d95e6cd.Highjinks@gmx.com> On Sun, 28 Mar 2010 09:58:04 +0000 Jay K wrote: > > I understand that userthreads don't scale across multiple processors. > > But testing out cvsup lately, things are noticably much much faster > > with user threads. At least cvsup. I haven't watched the compiler and such. > > > > > > - Jay In fact user threads, and better yet, Synchronous Threads, are in fact highly scalable when done right. This has been one point of contention for me regarding the current crop of "mature" programming languages. Everything is being shoehorned into the Asynchronous model. The problem is that the Asynchronous model is staring to break down. With more and more processor cores being added, the need for more and more locks and mutexes and semaphores is growing. Not only does this kill performance, it makes problems like deadlock and race conditions much more difficult to prevent. It used to be that user threads, cooperative threads, and Synchronous threads were seen as bottlenecks. That might have been the case back when we were dealing maybe four processors max. But now were looking, in some cases, at machines with four 4 core processors on a MB; and that number is expected to keep growing. Now Cooperative threads and Synchronous threads are outscaling Asynchronous threads, particularly in multimedia applications. The thing that is really needed now is language support and kernel support for the Synchronous model. Here are some reference links, for those who are curious... The Kusp project, which has support for Synchronous threads in the Linux Kernel. http://www.ittc.ku.edu/kurt/ A Wikipedia link on the Esterel programming language. http://en.wikipedia.org/wiki/Esterel Project COSA, which explores the issue a little more in depth. http://www.rebelscience.org/Cosas/COSA.htm The Timber compiler, which is all about Synchronous Programming. For those who care to experiment. http://timber-lang.org/ I realize the OP is about "user threads", but I think it might be good to open up a discussion about "user threads" and the subtopics therein. Modula might benefit from a formal definition of this stuff. IMHO. Anyone else share my interest? -- Chris From jay.krell at cornell.edu Mon Mar 29 03:59:39 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 29 Mar 2010 01:59:39 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <20100328170619.5d95e6cd.Highjinks@gmx.com> References: , <20100328170619.5d95e6cd.Highjinks@gmx.com> Message-ID: People need to know to reduce cross-thread sharing, to reduce lock contention. It is not impossible. There are various conflicting trends though, granted. Windows has long had fibers, but they are pretty impossible to use -- you need your own locking mechanisms, which interact with your own scheduler. There is now "user mode scheduling" in Windows 7 that is easier to use, and the "concurrent runtime" makes it easier to use, along with helping reduce sharing. You should sort of "map/reduce" sorts of things, where each thread can do some of the work independently and there is one last mop-up or sweep pass to collect the data. Consider sorting large data sets for example. You can break up the data into N pieces, sort of them independently on N threads, and then do one last single threaded pass to merge them. "Cloud computing" pushes people toward models of very coarse grained parallelism -- multiple machines, not even the "old coarse grained" multiple processes on the same machine. I think profiling is maybe in order to understand cvsup performance with user threads vs. kernel threads. Usually I ignore performance but the difference is quite noticable in my limited testing. Personally I've only been convinced that kernel threads are generally best and simplest. There are tradeoffs though. Having one mondo scheduler puts a lot of pressure on that one scheduler to figure out things for everyone, whereas within each program there is probably local information that could provide for better scheduling. Basically, taking locks or waiting on objects (files, events, semaphores) is the only way to communicate to a scheduler. It helps a lot, but maybe is not always all that could be done. - Jay > From: Highjinks at gmx.com > To: m3devel at elegosoft.com > Date: Sun, 28 Mar 2010 23:01:31 +0200 > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > On Sun, 28 Mar 2010 09:58:04 +0000 > Jay K wrote: > > > > > I understand that userthreads don't scale across multiple processors. > > > > But testing out cvsup lately, things are noticably much much faster > > > > with user threads. At least cvsup. I haven't watched the compiler and such. > > > > > > > > > > > > - Jay > > In fact user threads, and better yet, Synchronous Threads, are in fact highly scalable when done right. > > This has been one point of contention for me regarding the current crop of "mature" programming languages. Everything is being shoehorned into the Asynchronous model. > > The problem is that the Asynchronous model is staring to break down. With more and more processor cores being added, the need for more and more locks and mutexes and semaphores is growing. Not only does this kill performance, it makes problems like deadlock and race conditions much more difficult to prevent. > > It used to be that user threads, cooperative threads, and Synchronous threads were seen as bottlenecks. That might have been the case back when we were dealing maybe four processors max. But now were looking, in some cases, at machines with four 4 core processors on a MB; and that number is expected to keep growing. Now Cooperative threads and Synchronous threads are outscaling Asynchronous threads, particularly in multimedia applications. > > The thing that is really needed now is language support and kernel support for the Synchronous model. > > Here are some reference links, for those who are curious... > > The Kusp project, which has support for Synchronous threads in the Linux Kernel. > http://www.ittc.ku.edu/kurt/ > > A Wikipedia link on the Esterel programming language. > http://en.wikipedia.org/wiki/Esterel > > Project COSA, which explores the issue a little more in depth. > http://www.rebelscience.org/Cosas/COSA.htm > > The Timber compiler, which is all about Synchronous Programming. For those who care to experiment. > http://timber-lang.org/ > > I realize the OP is about "user threads", but I think it might be good to open up a discussion about "user threads" and the subtopics therein. Modula might benefit from a formal definition of this stuff. IMHO. > > Anyone else share my interest? > > -- > Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Mar 29 04:06:20 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 29 Mar 2010 02:06:20 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <20100328170619.5d95e6cd.Highjinks@gmx.com> References: , <20100328170619.5d95e6cd.Highjinks@gmx.com> Message-ID: http://www.ittc.ku.edu/kurt/ wow, it implies the Linux kernel isn't preemptible...but I did a bit more research..as of 2.6 it is preemtible, configurable, but I don't know what the default is. I failed to mention Erlang, which also pushes process-level concurrency, and much of it. You still need some of locks in these systems, granted. Like file locks. - Jay From: jay.krell at cornell.edu To: highjinks at gmx.com; m3devel at elegosoft.com Subject: RE: [M3devel] userthreads vs. pthreads performance? Date: Mon, 29 Mar 2010 01:59:39 +0000 People need to know to reduce cross-thread sharing, to reduce lock contention. It is not impossible. There are various conflicting trends though, granted. Windows has long had fibers, but they are pretty impossible to use -- you need your own locking mechanisms, which interact with your own scheduler. There is now "user mode scheduling" in Windows 7 that is easier to use, and the "concurrent runtime" makes it easier to use, along with helping reduce sharing. You should sort of "map/reduce" sorts of things, where each thread can do some of the work independently and there is one last mop-up or sweep pass to collect the data. Consider sorting large data sets for example. You can break up the data into N pieces, sort of them independently on N threads, and then do one last single threaded pass to merge them. "Cloud computing" pushes people toward models of very coarse grained parallelism -- multiple machines, not even the "old coarse grained" multiple processes on the same machine. I think profiling is maybe in order to understand cvsup performance with user threads vs. kernel threads. Usually I ignore performance but the difference is quite noticable in my limited testing. Personally I've only been convinced that kernel threads are generally best and simplest. There are tradeoffs though. Having one mondo scheduler puts a lot of pressure on that one scheduler to figure out things for everyone, whereas within each program there is probably local information that could provide for better scheduling. Basically, taking locks or waiting on objects (files, events, semaphores) is the only way to communicate to a scheduler. It helps a lot, but maybe is not always all that could be done. - Jay > From: Highjinks at gmx.com > To: m3devel at elegosoft.com > Date: Sun, 28 Mar 2010 23:01:31 +0200 > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > On Sun, 28 Mar 2010 09:58:04 +0000 > Jay K wrote: > > > > > I understand that userthreads don't scale across multiple processors. > > > > But testing out cvsup lately, things are noticably much much faster > > > > with user threads. At least cvsup. I haven't watched the compiler and such. > > > > > > > > > > > > - Jay > > In fact user threads, and better yet, Synchronous Threads, are in fact highly scalable when done right. > > This has been one point of contention for me regarding the current crop of "mature" programming languages. Everything is being shoehorned into the Asynchronous model. > > The problem is that the Asynchronous model is staring to break down. With more and more processor cores being added, the need for more and more locks and mutexes and semaphores is growing. Not only does this kill performance, it makes problems like deadlock and race conditions much more difficult to prevent. > > It used to be that user threads, cooperative threads, and Synchronous threads were seen as bottlenecks. That might have been the case back when we were dealing maybe four processors max. But now were looking, in some cases, at machines with four 4 core processors on a MB; and that number is expected to keep growing. Now Cooperative threads and Synchronous threads are outscaling Asynchronous threads, particularly in multimedia applications. > > The thing that is really needed now is language support and kernel support for the Synchronous model. > > Here are some reference links, for those who are curious... > > The Kusp project, which has support for Synchronous threads in the Linux Kernel. > http://www.ittc.ku.edu/kurt/ > > A Wikipedia link on the Esterel programming language. > http://en.wikipedia.org/wiki/Esterel > > Project COSA, which explores the issue a little more in depth. > http://www.rebelscience.org/Cosas/COSA.htm > > The Timber compiler, which is all about Synchronous Programming. For those who care to experiment. > http://timber-lang.org/ > > I realize the OP is about "user threads", but I think it might be good to open up a discussion about "user threads" and the subtopics therein. Modula might benefit from a formal definition of this stuff. IMHO. > > Anyone else share my interest? > > -- > Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Mar 29 05:15:41 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 29 Mar 2010 03:15:41 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269803697.3671.19.camel@faramir.m3w.org> References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> Message-ID: > Getting thread locals should not require a kernel call Indeed, on Linux/x86 it does not, looks pretty ok: 00000380 <__pthread_getspecific>: 380: 55 push %ebp 381: 89 e5 mov %esp,%ebp 383: 8b 55 08 mov 0x8(%ebp),%edx 386: 81 fa ff 03 00 00 cmp $0x3ff,%edx 38c: 76 04 jbe 392 <__pthread_getspecific+0x12> 38e: 5d pop %ebp 38f: 31 c0 xor %eax,%eax 391: c3 ret 392: 89 d0 mov %edx,%eax 394: c1 e8 05 shr $0x5,%eax 397: 8d 0c 85 1c 01 00 00 lea 0x11c(,%eax,4),%ecx 39e: 65 8b 01 mov %gs:(%ecx),%eax 3a1: 85 c0 test %eax,%eax 3a3: 74 e9 je 38e <__pthread_getspecific+0xe> 3a5: 8b 04 d5 00 00 00 00 mov 0x0(,%edx,8),%eax 3ac: 85 c0 test %eax,%eax 3ae: 74 de je 38e <__pthread_getspecific+0xe> 3b0: 65 8b 01 mov %gs:(%ecx),%eax 3b3: 83 e2 1f and $0x1f,%edx 3b6: 8b 04 90 mov (%eax,%edx,4),%eax 3b9: 5d pop %ebp 3ba: c3 ret > Entering an uncontended pthread mutex should not be expensive Linux/x86: 00001020 <__pthread_self>: 1020: 55 push %ebp 1021: 89 e5 mov %esp,%ebp 1023: 65 a1 50 00 00 00 mov %gs:0x50,%eax 1029: 5d pop %ebp 102a: c3 ret 102b: 90 nop 102c: 8d 74 26 00 lea 0x0(%esi),%esi pretty lame, five instructions were only two are needed. 000004f0 <__pthread_mutex_lock>: .. too much to read through..but I think no kernel call.. - Jay From: jay.krell at cornell.edu To: dragisha at m3w.org; mika at async.async.caltech.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] userthreads vs. pthreads performance? Date: Sun, 28 Mar 2010 20:46:01 +0000 O(1) scheduling is not a new idea. Just look at NT and probably Solaris and probably all the other non-free systems (AIX, Irix, HP-UX, Tru64, VMS, etc.) Getting thread locals should not require a kernel call. It doesn't on NT. We can optimize this somewhat on most systems with __thread. I had that in briefly. Entering an uncontended pthread mutex should not be expensive -- at least no kernel call, but granted a call and atomic op. Two calls because of the C layer. But user threads pay for a call too of course. Maybe I should profile some of this.. - Jay > From: dragisha at m3w.org > To: mika at async.async.caltech.edu > Date: Sun, 28 Mar 2010 21:14:57 +0200 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > I remember reading (long time ago) about how these (FUTEXes) are > efficient in LINUX... Can I have your test code to try? > > On Sun, 2010-03-28 at 12:11 -0700, Mika Nystrom wrote: > > Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... > > > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > > >Which platform? > > > > > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > > >> Yep, sounds right. > > >> > > >> I was profiling some other thread-using code that slowed down > > >> enormously > > >> because of pthreads and it turned out the program was spending ~95% > > >> of its time in accessing the thread locals via one of the pthread_ > > >> functions. > > >> (The overhead of entering the kernel.) > > >-- > > >Dragi?a Duri? > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Mar 29 10:24:47 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 29 Mar 2010 08:24:47 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269803697.3671.19.camel@faramir.m3w.org> References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> Message-ID: > We can optimize this somewhat on most systems with __thread. I had that in briefly I did a little bit of testing here. __thread takes a few forms, depending on which of -fPIC and -shared you use. Once you do throw in -fPIC and -shared, I have found __thread to be significantly slower on Solaris/sparc and Linux/powerpc32, slower or a wash on Linux/amd64, and twice as fast as pthread_getspecific on Linux/x86. I doesn't appear supported at all on Darwin, though pthread_getspecific are very fast there (albeit not inlined). I didn't test *BSD. My testing was not very scientific. However -fPIC and/or -shared imply a function call to access __thread variables. That's probably the big factor. Without -fPIC/-shared, there is no function call. If you are going to access them multiple times in a function then there is a probably an optimization to be had -- caching their address. If the variables are larger than a pointer, probably then also an optimization to be had. We could do that first thing for certain -- PushFrame could return the address and PopFrame would be much faster. However another angle here is to eliminate PushFrame/PopFrame, by using libunwind. I think we should look into libunwind for the next release. The thread locals will (mostly) remain but accesses to them greatly decline. We compile everything -fPIC and -shared. Some systems (libtool) compile things once that way and once not, providing pairs of libraries, depending on intended use. I should point out also that userthreads have been greatly deoptimized in the current tree (by me, Tony approved), because they used to inline PushFrame/PopFrame, but they don't any longer. (Historically on NT, __declspec(thread) only worked in code in the .exe or statically loaded by the .exe -- that is, not in .dlls loaded with LoadLibrary. However that limitation was removed in Vista. I expect __declspec(thread) is much faster than TlsGetValue, but I assume we'll support pre-Vista for a while longer so not interesting..) - Jay From: jay.krell at cornell.edu To: dragisha at m3w.org; mika at async.async.caltech.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] userthreads vs. pthreads performance? Date: Mon, 29 Mar 2010 03:15:41 +0000 > Getting thread locals should not require a kernel call Indeed, on Linux/x86 it does not, looks pretty ok: 00000380 <__pthread_getspecific>: 380: 55 push %ebp 381: 89 e5 mov %esp,%ebp 383: 8b 55 08 mov 0x8(%ebp),%edx 386: 81 fa ff 03 00 00 cmp $0x3ff,%edx 38c: 76 04 jbe 392 <__pthread_getspecific+0x12> 38e: 5d pop %ebp 38f: 31 c0 xor %eax,%eax 391: c3 ret 392: 89 d0 mov %edx,%eax 394: c1 e8 05 shr $0x5,%eax 397: 8d 0c 85 1c 01 00 00 lea 0x11c(,%eax,4),%ecx 39e: 65 8b 01 mov %gs:(%ecx),%eax 3a1: 85 c0 test %eax,%eax 3a3: 74 e9 je 38e <__pthread_getspecific+0xe> 3a5: 8b 04 d5 00 00 00 00 mov 0x0(,%edx,8),%eax 3ac: 85 c0 test %eax,%eax 3ae: 74 de je 38e <__pthread_getspecific+0xe> 3b0: 65 8b 01 mov %gs:(%ecx),%eax 3b3: 83 e2 1f and $0x1f,%edx 3b6: 8b 04 90 mov (%eax,%edx,4),%eax 3b9: 5d pop %ebp 3ba: c3 ret > Entering an uncontended pthread mutex should not be expensive Linux/x86: 00001020 <__pthread_self>: 1020: 55 push %ebp 1021: 89 e5 mov %esp,%ebp 1023: 65 a1 50 00 00 00 mov %gs:0x50,%eax 1029: 5d pop %ebp 102a: c3 ret 102b: 90 nop 102c: 8d 74 26 00 lea 0x0(%esi),%esi pretty lame, five instructions were only two are needed. 000004f0 <__pthread_mutex_lock>: .. too much to read through..but I think no kernel call.. - Jay From: jay.krell at cornell.edu To: dragisha at m3w.org; mika at async.async.caltech.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] userthreads vs. pthreads performance? Date: Sun, 28 Mar 2010 20:46:01 +0000 O(1) scheduling is not a new idea. Just look at NT and probably Solaris and probably all the other non-free systems (AIX, Irix, HP-UX, Tru64, VMS, etc.) Getting thread locals should not require a kernel call. It doesn't on NT. We can optimize this somewhat on most systems with __thread. I had that in briefly. Entering an uncontended pthread mutex should not be expensive -- at least no kernel call, but granted a call and atomic op. Two calls because of the C layer. But user threads pay for a call too of course. Maybe I should profile some of this.. - Jay > From: dragisha at m3w.org > To: mika at async.async.caltech.edu > Date: Sun, 28 Mar 2010 21:14:57 +0200 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > I remember reading (long time ago) about how these (FUTEXes) are > efficient in LINUX... Can I have your test code to try? > > On Sun, 2010-03-28 at 12:11 -0700, Mika Nystrom wrote: > > Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... > > > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > > >Which platform? > > > > > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > > >> Yep, sounds right. > > >> > > >> I was profiling some other thread-using code that slowed down > > >> enormously > > >> because of pthreads and it turned out the program was spending ~95% > > >> of its time in accessing the thread locals via one of the pthread_ > > >> functions. > > >> (The overhead of entering the kernel.) > > >-- > > >Dragi?a Duri? > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Mon Mar 29 11:38:18 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Mon, 29 Mar 2010 11:38:18 +0200 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: ,<1269771357.3671.7.camel@faramir.m3w.org> ,<20100328162424.670F91A20B7@async.async.caltech.edu> ,<9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> ,<1269801143.3671.17.camel@faramir.m3w.org> ,<20100328185740.38AFA1A20B9@async.async.caltech.edu> ,<1269802924.3671.18.camel@faramir.m3w.org> ,<20100328191116.CD9C71A20BB@async.async.caltech.edu> ,<1269803697.3671.19.camel@faramir.m3w.org> Message-ID: <1269855498.2369.1.camel@faramir.m3w.org> I can be wrong, but what FUTEX operations on Linux can have with pthread_getspecific? Operations are made on FUTEX, we know which and where, no need to ask for pthread_getspecific. Are we still talking about poor lock_mutex performance? On Mon, 2010-03-29 at 08:24 +0000, Jay K wrote: > Once you do throw in -fPIC and -shared, I have found __thread to be > significantly slower on Solaris/sparc and Linux/powerpc32, slower or > a wash on Linux/amd64, and twice as fast as pthread_getspecific on > Linux/x86. > I doesn't appear supported at all on Darwin, though > pthread_getspecific are very fast there (albeit not inlined). > I didn't test *BSD. -- Dragi?a Duri? From jay.krell at cornell.edu Mon Mar 29 15:10:36 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 29 Mar 2010 13:10:36 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269855498.2369.1.camel@faramir.m3w.org> References: , , <1269771357.3671.7.camel@faramir.m3w.org>, , <20100328162424.670F91A20B7@async.async.caltech.edu>, , <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, , <1269801143.3671.17.camel@faramir.m3w.org>, , <20100328185740.38AFA1A20B9@async.async.caltech.edu>, , <1269802924.3671.18.camel@faramir.m3w.org>, , <20100328191116.CD9C71A20BB@async.async.caltech.edu>, , <1269803697.3671.19.camel@faramir.m3w.org>, , <1269855498.2369.1.camel@faramir.m3w.org> Message-ID: Original was more general pthread vs. userthread and I thought pthread_getspecific was also indicted. - Jay > From: dragisha at m3w.org > To: jay.krell at cornell.edu > Date: Mon, 29 Mar 2010 11:38:18 +0200 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > I can be wrong, but what FUTEX operations on Linux can have with > pthread_getspecific? > > Operations are made on FUTEX, we know which and where, no need to ask > for pthread_getspecific. > > Are we still talking about poor lock_mutex performance? > > > On Mon, 2010-03-29 at 08:24 +0000, Jay K wrote: > > Once you do throw in -fPIC and -shared, I have found __thread to be > > significantly slower on Solaris/sparc and Linux/powerpc32, slower or > > a wash on Linux/amd64, and twice as fast as pthread_getspecific on > > Linux/x86. > > I doesn't appear supported at all on Darwin, though > > pthread_getspecific are very fast there (albeit not inlined). > > I didn't test *BSD. > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 29 16:42:17 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 29 Mar 2010 10:42:17 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> Message-ID: Guys, You are looking in the wrong place. The whole good thing about having a language-defined thread model is that we get to implement the thread primitives in the language run-time, and we can take advantage of language-specific semantics. There is no requirement that a Modula-3 mutex even map to a pthread_mutex_t. Java monitors are implemented in modern JVMs so that lock/unlock in the common case doesn't require any calls or any atomic operations! We can do much the same for Modula-3. My group at Purdue has done a fair amount of work on this in the context of Java, and when I can find the time I was hoping to do the same for Modula-3. The only requirement for M3 was that we have proper support for atomic ops in the language upon which to build the synchronisation primitives. We are close to having this now. Let me sketch a design: The common case is that a mutex is only ever manipulated by one thread (i.e., never shared), in which case it suffices to "bias" the mutex to that thread. Locking/unlocking is simply a matter of checking to see that the bias belongs to the current thread. No need for an atomic operation. If the thread already has the bias then locking/unlocking is simply a matter of setting a bit in the mutex. If another thread comes along and needs to lock the mutex then it must first revoke the bias of the owning thread. This can be expensive (assuming it occurs infrequently) and in our case probably means stopping the thread having the bias, revoking the bias, then restarting it. Another case is when a mutex is locked/unlocked by multiple threads but there is never contention (i.e., no thread tries to acquire while another thread holds). In this case we never need a wait queue for the mutex so we can simply store the lock owner in the mutex and test using atomic ops. Spinning is often useful here to avoid needing to inflate if contention ever does arise: if the thread holding the lock gets out of the mutex quickly then the spinner can move in quickly. After some number of spins we generally need to inflate the lock to allocate a wait queue (to avoiding hogging the processor). Finally, the case where many threads are contending on the inflated lock (with wait queue). The only question now is when to deflate. Our current heuristic is to deflate when the last thread releases the lock and notices that there are no other threads waiting. This seems to work well in practice, but of course there are pathological cases. Note that in no case have I mentioned the need for a pthread_mutex (though pthread locks/conditions are used to manage threads that must block on a Java quit queue). We ought to be able to do much the same in Modula-3. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 29 Mar 2010, at 04:24, Jay K wrote: > > We can optimize this somewhat on most systems with __thread. I had that in briefly > > > I did a little bit of testing here. > > > __thread takes a few forms, depending on which of -fPIC and -shared you use. > > Once you do throw in -fPIC and -shared, I have found __thread to be significantly slower on Solaris/sparc and Linux/powerpc32, slower or a wash on Linux/amd64, and twice as fast as pthread_getspecific on Linux/x86. > I doesn't appear supported at all on Darwin, though pthread_getspecific are very fast there (albeit not inlined). > I didn't test *BSD. > My testing was not very scientific. > However -fPIC and/or -shared imply a function call to access __thread variables. > That's probably the big factor. Without -fPIC/-shared, there is no function call. > > > If you are going to access them multiple times in a function then there is a probably an optimization to be had -- caching their address. If the variables are larger than a pointer, probably then also an optimization to be had. > > > We could do that first thing for certain -- PushFrame could return the address and PopFrame would be much faster. > However another angle here is to eliminate PushFrame/PopFrame, by using libunwind. I think we should look into libunwind for the next release. The thread locals will (mostly) remain but accesses to them greatly decline. > > > We compile everything -fPIC and -shared. > Some systems (libtool) compile things once that way and once not, providing pairs of libraries, depending on intended use. > > > I should point out also that userthreads have been greatly deoptimized in the current tree (by me, Tony approved), because they used to inline PushFrame/PopFrame, but they don't any longer. > > > (Historically on NT, __declspec(thread) only worked in code in the .exe or statically loaded by the .exe -- that is, not in .dlls loaded with LoadLibrary. However that limitation was removed in Vista. I expect __declspec(thread) is much faster than TlsGetValue, but I assume we'll support pre-Vista for a while longer so not interesting..) > > > - Jay > > > From: jay.krell at cornell.edu > To: dragisha at m3w.org; mika at async.async.caltech.edu > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] userthreads vs. pthreads performance? > Date: Mon, 29 Mar 2010 03:15:41 +0000 > > > Getting thread locals should not require a kernel call > > Indeed, on Linux/x86 it does not, looks pretty ok: > > 00000380 <__pthread_getspecific>: > 380: 55 push %ebp > 381: 89 e5 mov %esp,%ebp > 383: 8b 55 08 mov 0x8(%ebp),%edx > 386: 81 fa ff 03 00 00 cmp $0x3ff,%edx > 38c: 76 04 jbe 392 <__pthread_getspecific+0x12> > 38e: 5d pop %ebp > 38f: 31 c0 xor %eax,%eax > 391: c3 ret > > 392: 89 d0 mov %edx,%eax > 394: c1 e8 05 shr $0x5,%eax > 397: 8d 0c 85 1c 01 00 00 lea 0x11c(,%eax,4),%ecx > 39e: 65 8b 01 mov %gs:(%ecx),%eax > 3a1: 85 c0 test %eax,%eax > 3a3: 74 e9 je 38e <__pthread_getspecific+0xe> > 3a5: 8b 04 d5 00 00 00 00 mov 0x0(,%edx,8),%eax > 3ac: 85 c0 test %eax,%eax > 3ae: 74 de je 38e <__pthread_getspecific+0xe> > 3b0: 65 8b 01 mov %gs:(%ecx),%eax > 3b3: 83 e2 1f and $0x1f,%edx > 3b6: 8b 04 90 mov (%eax,%edx,4),%eax > 3b9: 5d pop %ebp > 3ba: c3 ret > > > > Entering an uncontended pthread mutex should not be expensive > > Linux/x86: > > 00001020 <__pthread_self>: > 1020: 55 push %ebp > 1021: 89 e5 mov %esp,%ebp > 1023: 65 a1 50 00 00 00 mov %gs:0x50,%eax > 1029: 5d pop %ebp > 102a: c3 ret > 102b: 90 nop > 102c: 8d 74 26 00 lea 0x0(%esi),%esi > > > pretty lame, five instructions were only two are needed. > > > 000004f0 <__pthread_mutex_lock>: > > .. too much to read through..but I think no kernel call.. > > - Jay > > From: jay.krell at cornell.edu > To: dragisha at m3w.org; mika at async.async.caltech.edu > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] userthreads vs. pthreads performance? > Date: Sun, 28 Mar 2010 20:46:01 +0000 > > O(1) scheduling is not a new idea. Just look at NT and probably Solaris and probably all the other non-free systems (AIX, Irix, HP-UX, Tru64, VMS, etc.) > > Getting thread locals should not require a kernel call. It doesn't on NT. We can optimize this somewhat on most systems with __thread. I had that in briefly. > > Entering an uncontended pthread mutex should not be expensive -- at least no kernel call, but granted a call and atomic op. Two calls because of the C layer. > But user threads pay for a call too of course. > > Maybe I should profile some of this.. > > - Jay > > > From: dragisha at m3w.org > > To: mika at async.async.caltech.edu > > Date: Sun, 28 Mar 2010 21:14:57 +0200 > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > > > I remember reading (long time ago) about how these (FUTEXes) are > > efficient in LINUX... Can I have your test code to try? > > > > On Sun, 2010-03-28 at 12:11 -0700, Mika Nystrom wrote: > > > Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... > > > > > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > > > >Which platform? > > > > > > > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > > > >> Yep, sounds right. > > > >> > > > >> I was profiling some other thread-using code that slowed down > > > >> enormously > > > >> because of pthreads and it turned out the program was spending ~95% > > > >> of its time in accessing the thread locals via one of the pthread_ > > > >> functions. > > > >> (The overhead of entering the kernel.) > > > >-- > > > >Dragi?a Duri? > > -- > > Dragi?a Duri? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 29 16:43:21 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 29 Mar 2010 10:43:21 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269855498.2369.1.camel@faramir.m3w.org> References: , <1269771357.3671.7.camel@faramir.m3w.org> , <20100328162424.670F91A20B7@async.async.caltech.edu> , <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> , <1269801143.3671.17.camel@faramir.m3w.org> , <20100328185740.38AFA1A20B9@async.async.caltech.edu> , <1269802924.3671.18.camel@faramir.m3w.org> , <20100328191116.CD9C71A20BB@async.async.caltech.edu> , <1269803697.3671.19.camel@faramir.m3w.org> <1269855498.2369.1.camel@faramir.m3w.org> Message-ID: PS. Our Java work shows that futexes don't really win against the bias/thin/fat lock scheme I sketched previously. On 29 Mar 2010, at 05:38, Dragi?a Duri? wrote: > I can be wrong, but what FUTEX operations on Linux can have with > pthread_getspecific? > > Operations are made on FUTEX, we know which and where, no need to ask > for pthread_getspecific. > > Are we still talking about poor lock_mutex performance? > > > On Mon, 2010-03-29 at 08:24 +0000, Jay K wrote: >> Once you do throw in -fPIC and -shared, I have found __thread to be >> significantly slower on Solaris/sparc and Linux/powerpc32, slower or >> a wash on Linux/amd64, and twice as fast as pthread_getspecific on >> Linux/x86. >> I doesn't appear supported at all on Darwin, though >> pthread_getspecific are very fast there (albeit not inlined). >> I didn't test *BSD. > -- > Dragi?a Duri? From rodney_bates at lcwb.coop Mon Mar 29 17:01:04 2010 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 29 Mar 2010 10:01:04 -0500 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> Message-ID: <4BB0C0B0.3010108@lcwb.coop> Tony Hosking wrote: > Guys, > > You are looking in the wrong place. The whole good thing about having a > language-defined thread model is that we get to implement the thread > primitives in the language run-time, and we can take advantage of > language-specific semantics. There is no requirement that a Modula-3 > mutex even map to a pthread_mutex_t. Java monitors are implemented in > modern JVMs so that lock/unlock in the common case doesn't require any > calls or any atomic operations! We can do much the same for Modula-3. > My group at Purdue has done a fair amount of work on this in the > context of Java, and when I can find the time I was hoping to do the > same for Modula-3. The only requirement for M3 was that we have proper > support for atomic ops in the language upon which to build the > synchronisation primitives. We are close to having this now. Let me > sketch a design: > > The common case is that a mutex is only ever manipulated by one thread Do you mean _almost_ only ever? -----^ Otherwise, why would it have been coded with a mutex at all? > (i.e., never shared), in which case it suffices to "bias" the mutex to > that thread. Locking/unlocking is simply a matter of checking to see > that the bias belongs to the current thread. No need for an atomic > operation. If the thread already has the bias then locking/unlocking is > simply a matter of setting a bit in the mutex. If another thread comes > along and needs to lock the mutex then it must first revoke the bias of > the owning thread. This can be expensive (assuming it occurs > infrequently) and in our case probably means stopping the thread having > the bias, revoking the bias, then restarting it. > > Another case is when a mutex is locked/unlocked by multiple threads but > there is never contention (i.e., no thread tries to acquire while -----------^ and _almost_ never? > another thread holds). In this case we never need a wait queue for the > mutex so we can simply store the lock owner in the mutex and test using > atomic ops. Spinning is often useful here to avoid needing to inflate > if contention ever does arise: if the thread holding the lock gets out > of the mutex quickly then the spinner can move in quickly. After some > number of spins we generally need to inflate the lock to allocate a wait > queue (to avoiding hogging the processor). > > Finally, the case where many threads are contending on the inflated lock > (with wait queue). The only question now is when to deflate. Our > current heuristic is to deflate when the last thread releases the lock > and notices that there are no other threads waiting. This seems to work > well in practice, but of course there are pathological cases. So does the implementation dynamically detect which case holds and choose between the schemes? What are the criteria? > > Note that in no case have I mentioned the need for a pthread_mutex > (though pthread locks/conditions are used to manage threads that must > block on a Java quit queue). > > We ought to be able to do much the same in Modula-3. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > From mika at async.async.caltech.edu Mon Mar 29 17:20:25 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Mon, 29 Mar 2010 08:20:25 -0700 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <4BB0C0B0.3010108@lcwb.coop> References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> <4BB0C0B0.3010108@lcwb.coop> Message-ID: <20100329152025.630591A20C2@async.async.caltech.edu> "Rodney M. Bates" writes: > ... >> >> The common case is that a mutex is only ever manipulated by one thread >Do you mean _almost_ only ever? -----^ Otherwise, why would it have been >coded with a mutex at all? ... In a single-threaded program.... lots of mutexes hiding in various software libraries (e.g., Rd/Wr)... Mika From hosking at cs.purdue.edu Mon Mar 29 19:26:28 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 29 Mar 2010 13:26:28 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <4BB0C0B0.3010108@lcwb.coop> References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> <4BB0C0B0.3010108@lcwb.coop> Message-ID: <3716771F-D63C-4ABC-8977-3B7F509BEB88@cs.purdue.edu> On 29 Mar 2010, at 11:01, Rodney M. Bates wrote: > > > Tony Hosking wrote: >> Guys, >> You are looking in the wrong place. The whole good thing about having a language-defined thread model is that we get to implement the thread primitives in the language run-time, and we can take advantage of language-specific semantics. There is no requirement that a Modula-3 mutex even map to a pthread_mutex_t. Java monitors are implemented in modern JVMs so that lock/unlock in the common case doesn't require any calls or any atomic operations! We can do much the same for Modula-3. My group at Purdue has done a fair amount of work on this in the context of Java, and when I can find the time I was hoping to do the same for Modula-3. The only requirement for M3 was that we have proper support for atomic ops in the language upon which to build the synchronisation primitives. We are close to having this now. Let me sketch a design: >> The common case is that a mutex is only ever manipulated by one thread > Do you mean _almost_ only ever? -----^ Otherwise, why would it have been > coded with a mutex at all? You might have a thread-safe library that is used by a single-thread application. >> (i.e., never shared), in which case it suffices to "bias" the mutex to that thread. Locking/unlocking is simply a matter of checking to see that the bias belongs to the current thread. No need for an atomic operation. If the thread already has the bias then locking/unlocking is simply a matter of setting a bit in the mutex. If another thread comes along and needs to lock the mutex then it must first revoke the bias of the owning thread. This can be expensive (assuming it occurs infrequently) and in our case probably means stopping the thread having the bias, revoking the bias, then restarting it. >> Another case is when a mutex is locked/unlocked by multiple threads but there is never contention (i.e., no thread tries to acquire while > -----------^ and _almost_ never? > >> another thread holds). In this case we never need a wait queue for the mutex so we can simply store the lock owner in the mutex and test using atomic ops. Spinning is often useful here to avoid needing to inflate if contention ever does arise: if the thread holding the lock gets out of the mutex quickly then the spinner can move in quickly. After some number of spins we generally need to inflate the lock to allocate a wait queue (to avoiding hogging the processor). >> Finally, the case where many threads are contending on the inflated lock (with wait queue). The only question now is when to deflate. Our current heuristic is to deflate when the last thread releases the lock and notices that there are no other threads waiting. This seems to work well in practice, but of course there are pathological cases. > > So does the implementation dynamically detect which case holds and > choose between the schemes? What are the criteria? Short answer, yes. Criteria much as I described. The only trick thing is choosing when to deflate. For more on this topic take a look also at: http://blogs.azulsystems.com/cliff/. Azul independently developed similar strategies. > >> Note that in no case have I mentioned the need for a pthread_mutex (though pthread locks/conditions are used to manage threads that must block on a Java quit queue). >> We ought to be able to do much the same in Modula-3. >> Antony Hosking | Associate Professor | Computer Science | Purdue University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> From dragisha at m3w.org Tue Mar 30 01:47:36 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Tue, 30 Mar 2010 01:47:36 +0200 Subject: [M3devel] user vs. kernel Message-ID: <1269906456.3029.4.camel@faramir.m3w.org> I am following this thread, and I have just few points to highlight: 1. battle user vs. kernel threads was finished long time ago. let's not pull back Modula-3 back where it was before last few years of development by doing things backward; 2. bias et al is nice thing, and I believe we will have it in no time; as compared to total development time of cm3. -- Dragi?a Duri? From jay.krell at cornell.edu Tue Mar 30 08:37:40 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 30 Mar 2010 06:37:40 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <3716771F-D63C-4ABC-8977-3B7F509BEB88@cs.purdue.edu> References: , , <1269771357.3671.7.camel@faramir.m3w.org>, , <20100328162424.670F91A20B7@async.async.caltech.edu>, , <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, , <1269801143.3671.17.camel@faramir.m3w.org>, , <20100328185740.38AFA1A20B9@async.async.caltech.edu>, , <1269802924.3671.18.camel@faramir.m3w.org>, , <20100328191116.CD9C71A20BB@async.async.caltech.edu>, , <1269803697.3671.19.camel@faramir.m3w.org> , , <4BB0C0B0.3010108@lcwb.coop>, <3716771F-D63C-4ABC-8977-3B7F509BEB88@cs.purdue.edu> Message-ID: > The only requirement for M3 was that we have proper support for > atomic ops in the language upon which to build the synchronisation > primitives. We are close to having this now. Or surely implement them in C.. Anyway, I believe m3back now (a few weeks) has parity with or exceeds the gcc backend. Depending on interpretation. The gcc backend only accepts some memory orders. m3back accepts them all and interprets them all as sequential. I have more testing and optimization to do, but I think it is all there and probably works. Optimization in particular is that add/sub/xor don't need to use compare/exchange/retry loops. And load/store could certainly be more efficient such as just using one exchange to do a store, instead of a fence on either side. I think it is also probably reasonable to defined variants that don't return the new value. That way or and and can be provided more efficiently -- without a compare/exchange/retry loop. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 30 14:51:34 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 30 Mar 2010 08:51:34 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: , , <1269771357.3671.7.camel@faramir.m3w.org>, , <20100328162424.670F91A20B7@async.async.caltech.edu>, , <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, , <1269801143.3671.17.camel@faramir.m3w.org>, , <20100328185740.38AFA1A20B9@async.async.caltech.edu>, , <1269802924.3671.18.camel@faramir.m3w.org>, , <20100328191116.CD9C71A20BB@async.async.caltech.edu>, , <1269803697.3671.19.camel@faramir.m3w.org> , , <4BB0C0B0.3010108@lcwb.coop>, <3716771F-D63C-4ABC-8977-3B7F509BEB88@cs.purdue.edu> Message-ID: On 30 Mar 2010, at 02:37, Jay K wrote: > > The only requirement for M3 was that we have proper support for > > atomic ops in the language upon which to build the synchronisation > > primitives. We are close to having this now. > > > Or surely implement them in C.. We want them inline. > Anyway, I believe m3back now (a few weeks) has parity with or exceeds the gcc backend. > Depending on interpretation. > > > The gcc backend only accepts some memory orders. > > > m3back accepts them all and interprets them all as sequential. same as gcc. > > > I have more testing and optimization to do, but I think it is all there > and probably works. Optimization in particular is that add/sub/xor > don't need to use compare/exchange/retry loops. > > > And load/store could certainly be more efficient such as just > using one exchange to do a store, instead of a fence on either side. > > > I think it is also probably reasonable to defined variants that > don't return the new value. That way or and and can be provided > more efficiently -- without a compare/exchange/retry loop. > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 31 02:53:27 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 00:53:27 +0000 Subject: [M3devel] m3core/src/types/Unicode.m3 var to const Message-ID: This is right, right? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 1.txt URL: From hosking at cs.purdue.edu Wed Mar 31 04:39:25 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 30 Mar 2010 22:39:25 -0400 Subject: [M3devel] m3core/src/types/Unicode.m3 var to const In-Reply-To: References: Message-ID: <91EA5B0C-96CA-4C36-B9C1-DEF2BC5D29CB@cs.purdue.edu> Yes, I think that looks OK. The original is very weird M3 code! Also, can you now make it a safe module instead of UNSAFE? (No use of ADR anymore). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 30 Mar 2010, at 20:53, Jay K wrote: > This is right, right? > > - Jay > > > > > > <1.txt> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 31 05:19:57 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 03:19:57 +0000 Subject: [M3devel] m3core/src/types/Unicode.m3 var to const In-Reply-To: <91EA5B0C-96CA-4C36-B9C1-DEF2BC5D29CB@cs.purdue.edu> References: , <91EA5B0C-96CA-4C36-B9C1-DEF2BC5D29CB@cs.purdue.edu> Message-ID: "I think I can". Just use arrays and indices more and pointers to elements less/not at all. Granted, moving the division is a big speed deoptimization since it is no longer by a constant (unless compiler is quite smart and does "partial inlining", and NT386 compiler is not, very few compilers are smart enough to see this). It might be worth a) restoring that small part b) having two functions, one for 2-tuples, one for 3-tuples, or even c) surely we con't need to handcode binary search here in the first place there is some generic reusable version? But that is tangential to var vs. const. I was also considering adding: Int8, UInt8, Int16, UInt16, UInt32, UInt64. In particular, in m3back, I think I'll be using UInt16 in tables -- CodeView 4.0 format uses UINT16 to represent types (CodeView 5.0 uses UINT32). In reality due to padding/alignment, UINT8/16 probably have no space savings, but they will get range checks..but I'll also have to add a range check when I "allocate" a new type. "That" is "why" I was looking in this directory. - Jay From: hosking at cs.purdue.edu Date: Tue, 30 Mar 2010 22:39:25 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] m3core/src/types/Unicode.m3 var to const Yes, I think that looks OK. The original is very weird M3 code! Also, can you now make it a safe module instead of UNSAFE? (No use of ADR anymore). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 30 Mar 2010, at 20:53, Jay K wrote: This is right, right? - Jay <1.txt> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 31 06:41:14 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 04:41:14 +0000 Subject: [M3devel] m3back directions? Message-ID: I'm curious what, if anything, people are interested in in m3back? There are several mostly independent directions: - remove it; use the gcc backend or other (burg, llvm, generate C) - expand to support other targets, AMD64_*, including AMD64_NT m3objfile would need macho/elf support for non-NT - expand to generate good debugging information for Microsoft debuggers - various smaller/larger optimizations inlining? Inlining seems like the most lacking optimization and thinking about it, it doesn't actually seem that hard to do, at least a little bit. I'm thinking for now of working on the debugging information. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 31 10:02:48 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 08:02:48 +0000 Subject: [M3devel] generic binary search in standard Modula-3 library? In-Reply-To: References: , , <91EA5B0C-96CA-4C36-B9C1-DEF2BC5D29CB@cs.purdue.edu>, Message-ID: > c) surely we con't need to handcode binary search here in the first place there is some generic reusable version? The closest precedent I can find is: http://modula3.elegosoft.com/cm3/doc/help/gen_html/libm3/src/sort/ArraySort.ig.html so I propose *something like*: m3core/src/binarysearch/BinarySearch.ig. PROCEDURE BinarySearch(READONLY key: Elem.T; READONLY a: ARRAY OF Elem.T; cmp := Elem.Compare): INTEGER; along with: PROCEDURE LowerBound(READONLY key: Elem.T; READONLY a: ARRAY OF Elem.T; cmp := Elem.Compare): INTEGER; PROCEDURE UpperBound(READONLY key: Elem.T; READONLY a: ARRAY OF Elem.T; cmp := Elem.Compare): INTEGER; PROCEDURE EqualRange(READONLY key: Elem.T; READONLY a: ARRAY OF Elem.T; VAR lower, upper: INTEGER; cmp := Elem.Compare); using -1 for not found? LowerBound, UpperBound, EqualRange are closely related to BinarySearch. When the array contains adjacent equal/equivalent elements, they let you find the first, last or first and last, whereas "regular" BinarySearch finds an arbitrary one. See the C++ STL. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 31 Mar 2010 03:19:57 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] m3core/src/types/Unicode.m3 var to const "I think I can". Just use arrays and indices more and pointers to elements less/not at all. Granted, moving the division is a big speed deoptimization since it is no longer by a constant (unless compiler is quite smart and does "partial inlining", and NT386 compiler is not, very few compilers are smart enough to see this). It might be worth a) restoring that small part b) having two functions, one for 2-tuples, one for 3-tuples, or even c) surely we con't need to handcode binary search here in the first place there is some generic reusable version? But that is tangential to var vs. const. I was also considering adding: Int8, UInt8, Int16, UInt16, UInt32, UInt64. In particular, in m3back, I think I'll be using UInt16 in tables -- CodeView 4.0 format uses UINT16 to represent types (CodeView 5.0 uses UINT32). In reality due to padding/alignment, UINT8/16 probably have no space savings, but they will get range checks..but I'll also have to add a range check when I "allocate" a new type. "That" is "why" I was looking in this directory. - Jay From: hosking at cs.purdue.edu Date: Tue, 30 Mar 2010 22:39:25 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] m3core/src/types/Unicode.m3 var to const Yes, I think that looks OK. The original is very weird M3 code! Also, can you now make it a safe module instead of UNSAFE? (No use of ADR anymore). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 30 Mar 2010, at 20:53, Jay K wrote: This is right, right? - Jay <1.txt> -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Wed Mar 31 11:37:31 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Wed, 31 Mar 2010 11:37:31 +0200 Subject: [M3devel] m3back directions? In-Reply-To: References: Message-ID: <1270028251.3114.4.camel@faramir.m3w.org> It is started job and useable right now. IMO - it's prolonged and thriving existence shows directions and maintains secure interface for multiple backends - LLVM for example. Thus said, my opinion is: a) keep it, and make it excellent working x86 backend alternative. b) AMD64, as natural successor to x86 - maybe, if it's not too much work. Even without it - m3back can probably work for many x86 appliances present - for years to come. c) debug - of course it's good idea. But, if it's alternative, one can debug her code using gcc, then switch to m3back build for productivity. d) nothing too big of optimizations, although inlining would be sweet. dd On Wed, 2010-03-31 at 04:41 +0000, Jay K wrote: > I'm curious what, if anything, people are interested in in m3back? > There are several mostly independent directions: > - remove it; use the gcc backend or other (burg, llvm, generate C) > - expand to support other targets, AMD64_*, including AMD64_NT > m3objfile would need macho/elf support for non-NT > - expand to generate good debugging information for Microsoft > debuggers > - various smaller/larger optimizations > inlining? Inlining seems like the most lacking optimization and > thinking about it, it doesn't actually seem that hard to do, at least > a little bit. -- Dragi?a Duri? From wagner at elegosoft.com Wed Mar 31 12:18:30 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Wed, 31 Mar 2010 12:18:30 +0200 Subject: [M3devel] m3back directions? In-Reply-To: References: Message-ID: <20100331121830.22q33okrokkwsgwk@mail.elegosoft.com> Quoting Jay K : > I'm curious what, if anything, people are interested in in m3back? > > There are several mostly independent directions: > > - remove it; use the gcc backend or other (burg, llvm, generate C) > > - expand to support other targets, AMD64_*, including AMD64_NT > > m3objfile would need macho/elf support for non-NT It would be great if we could use the integrated backend for other targets, too. Adding ELF support will be a lot of work, but it's probably worth it. Olaf > - expand to generate good debugging information for Microsoft debuggers > > - various smaller/larger optimizations > > inlining? Inlining seems like the most lacking optimization and > thinking about it, it doesn't actually seem that hard to do, at > least a little bit. > > > > > > I'm thinking for now of working on the debugging information. > > > > > > - Jay > > > From jay.krell at cornell.edu Wed Mar 31 13:01:17 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 11:01:17 +0000 Subject: [M3devel] m3back directions? In-Reply-To: <20100331121830.22q33okrokkwsgwk@mail.elegosoft.com> References: , <20100331121830.22q33okrokkwsgwk@mail.elegosoft.com> Message-ID: > remove it? Oops, did I say that? :) Well, we should definitely do some of the others if are going to keep it. :) I'm leaning toward Microsoft CodeView debugging information currently. It should be the easiest and provide a lot of value. Though it is actually in m3objfile. I forgot to mention: It is already I think very portable to all 32bit x86 platforms. The real work work would be in m3objfile. A little in m3back regarding calling conventions perhaps. Maybe a little around munging function names for "-shared -fPIC" too. Like appending "@plt" in function calls or somesuch. - Jay > Date: Wed, 31 Mar 2010 12:18:30 +0200 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] m3back directions? > > Quoting Jay K : > > > I'm curious what, if anything, people are interested in in m3back? > > > > There are several mostly independent directions: > > > > - remove it; use the gcc backend or other (burg, llvm, generate C) > > > > - expand to support other targets, AMD64_*, including AMD64_NT > > > > m3objfile would need macho/elf support for non-NT > > It would be great if we could use the integrated backend for other > targets, too. Adding ELF support will be a lot of work, but it's probably > worth it. > > Olaf > > > - expand to generate good debugging information for Microsoft debuggers > > > > - various smaller/larger optimizations > > > > inlining? Inlining seems like the most lacking optimization and > > thinking about it, it doesn't actually seem that hard to do, at > > least a little bit. > > > > > > > > > > > > I'm thinking for now of working on the debugging information. > > > > > > > > > > > > - Jay > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 31 13:15:08 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 11:15:08 +0000 Subject: [M3devel] changing m3middle/m3back to have multiple passes? Message-ID: I'm thinking m3back could use multiple passes. There a few reasons, things that would be easier/possible if it had them. I know adding passes will slow it down, but I suspect it will still be about as pleasantly fast as it is now. The optimizations I have in mind: inlining, including where function definitions come after the call I think inlining where the "small" functions come first is doable in pass, but not if uses come before calls. not saving unused registers This only a small optimization and could probably be done in one pass if m3objfile got a "memmove" operation. Well, I already have the optimization in somewhat, but it involves nop-ing out code instead of removing it completely. dead store removal, and then removing the resulting dead code to compute the value possibly better register allocation -- e.g. based on noticing which variables are used often, or based on how they are used, for example if a variable (or expression/temporary) ends serving a shift count, we should favor putting it in ecx up front, instead of moving it to ecx later, but we don't generally know this till too late As I understand, m3middle already has the notion of "recording" and "playing back" calls to the M3CG interface. That seems like a good starting point? Some optimizations can be made via "CG to CG" transforms. However I think in general I think the required design would include: - ability to add in "private operations"; maybe - ability to add "private data" to existing operations A very simple one is annotate functions with required frame size. This isn't currently known until the end of the function. - clearly, ability to remove operations Can/should we somehow augment m3middle in support of this line of thinking? Or is that just not the way to go? Each backend should have its own internal representation and loop over it? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 31 16:02:36 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 31 Mar 2010 10:02:36 -0400 Subject: [M3devel] m3back directions? In-Reply-To: References: Message-ID: <376BB9E6-AC9B-4110-AD22-83F8186BE5BF@cs.purdue.edu> On 31 Mar 2010, at 00:41, Jay K wrote: > I'm curious what, if anything, people are interested in in m3back? Simple. Fast. It is target-dependent so optimising there is misguided effort unless the opts are target-dependent (instruction selection, register allocation!). We don't have resources to maintain a sophisticated backend. Better to rely on other tools. There was a Linux version in pm3 that we should be able to model on. > There are several mostly independent directions: > - remove it; use the gcc backend or other (burg, llvm, generate C) > - expand to support other targets, AMD64_*, including AMD64_NT > m3objfile would need macho/elf support for non-NT > - expand to generate good debugging information for Microsoft debuggers > - various smaller/larger optimizations > inlining? Inlining seems like the most lacking optimization and thinking about it, it doesn't actually seem that hard to do, at least a little bit. > > > I'm thinking for now of working on the debugging information. > > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 31 18:08:38 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 31 Mar 2010 12:08:38 -0400 Subject: [M3devel] m3back directions? In-Reply-To: <20100331121830.22q33okrokkwsgwk@mail.elegosoft.com> References: <20100331121830.22q33okrokkwsgwk@mail.elegosoft.com> Message-ID: On 31 Mar 2010, at 06:18, wagner at elegosoft.com wrote: > Quoting Jay K : > >> I'm curious what, if anything, people are interested in in m3back? >> >> There are several mostly independent directions: >> >> - remove it; use the gcc backend or other (burg, llvm, generate C) >> >> - expand to support other targets, AMD64_*, including AMD64_NT >> >> m3objfile would need macho/elf support for non-NT > > It would be great if we could use the integrated backend for other > targets, too. Adding ELF support will be a lot of work, but it's probably > worth it. Have you looked at the pm3 Linux support? > > Olaf > >> - expand to generate good debugging information for Microsoft debuggers >> >> - various smaller/larger optimizations >> >> inlining? Inlining seems like the most lacking optimization and thinking about it, it doesn't actually seem that hard to do, at least a little bit. >> >> >> >> >> >> I'm thinking for now of working on the debugging information. >> >> >> >> >> >> - Jay >> >> >> > > From wagner at elegosoft.com Mon Mar 1 11:21:52 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 01 Mar 2010 11:21:52 +0100 Subject: [M3devel] Tinderbox AMD64_LINUX and errors in libm3 tests Message-ID: <20100301112152.v325as9n4c48coo0@mail.elegosoft.com> Hi, after some script merges from the release branch to the trunk and adaptation of the tinderbox scripts on birch this platform finally seems to build and test OK again. However, I noticed a compilation error in the libm3 tests: --- tests in /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX--- new source -> compiling PathnameTests.i3 new source -> compiling PathnameTests.m3 new source -> compiling Subr.i3 new source -> compiling TextSubrTbl.i3 new source -> compiling TextSubrTbl.m3 new source -> compiling OSTest.m3 "../src/OSTest.m3", line 298: incompatible types (i) 1 error encountered compilation failed => not building program "OSTest" performing pathname-tests... /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX Fatal Error: package build failed cm3 returned 2 Would anybody care to fix that? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Mon Mar 1 11:42:57 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 1 Mar 2010 10:42:57 +0000 Subject: [M3devel] Tinderbox AMD64_LINUX and errors in libm3 tests In-Reply-To: <20100301112152.v325as9n4c48coo0@mail.elegosoft.com> References: <20100301112152.v325as9n4c48coo0@mail.elegosoft.com> Message-ID: I only ever use m3-sys/m3ests. I haven't learned to use any of the other tests. Surely this is longint. I'll look/fix shortly, though we could really use more participation. - Jay > Date: Mon, 1 Mar 2010 11:21:52 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: [M3devel] Tinderbox AMD64_LINUX and errors in libm3 tests > > Hi, > > after some script merges from the release branch to the trunk and > adaptation of the tinderbox scripts on birch this platform finally > seems to build and test OK again. > > However, I noticed a compilation error in the libm3 tests: > > --- tests in > /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX--- > new source -> compiling PathnameTests.i3 > new source -> compiling PathnameTests.m3 > new source -> compiling Subr.i3 > new source -> compiling TextSubrTbl.i3 > new source -> compiling TextSubrTbl.m3 > new source -> compiling OSTest.m3 > "../src/OSTest.m3", line 298: incompatible types (i) > 1 error encountered > compilation failed => not building program "OSTest" > performing pathname-tests... > /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX > Fatal Error: package build failed > > cm3 returned 2 > > Would anybody care to fix that? > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Mon Mar 1 11:59:27 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 01 Mar 2010 11:59:27 +0100 Subject: [M3devel] Tinderbox AMD64_LINUX and errors in libm3 tests In-Reply-To: References: <20100301112152.v325as9n4c48coo0@mail.elegosoft.com> Message-ID: <20100301115927.xn0vc5tnggksgggc@mail.elegosoft.com> Quoting Jay K : > I only ever use m3-sys/m3ests. I haven't learned to use any of the > other tests. These tests are included in the package tests. Each package may have one or more associated test directories. The tests are run after the package has been built in the test-all-pkgs jobs, both in tinderbox http://www.modula3.com/cm3/logs/package-status.html http://www.modula3.com/cm3/logs/cm3-pkg-report-AMD64_LINUX-2010-03-01-03-31-05.html http://www.modula3.com/cm3/logs/cm3-pkg-report-FreeBSD4-2010-03-01-04-53-25.html ... and in Hudson, for example http://hudson.modula3.com:8080/job/cm3-test-all-pkgs-AMD64_LINUX/ http://hudson.modula3.com:8080/job/cm3-test-all-pkgs-PPC_DARWIN/ http://hudson.modula3.com:8080/job/cm3-test-all-pkgs-FreeBSD4/ ... > Surely this is longint. I'll look/fix shortly, though we could > really use more participation. Great. Thanks, Olaf > - Jay > >> Date: Mon, 1 Mar 2010 11:21:52 +0100 >> From: wagner at elegosoft.com >> To: m3devel at elegosoft.com >> Subject: [M3devel] Tinderbox AMD64_LINUX and errors in libm3 tests >> >> Hi, >> >> after some script merges from the release branch to the trunk and >> adaptation of the tinderbox scripts on birch this platform finally >> seems to build and test OK again. >> >> However, I noticed a compilation error in the libm3 tests: >> >> --- tests in >> /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX--- >> new source -> compiling PathnameTests.i3 >> new source -> compiling PathnameTests.m3 >> new source -> compiling Subr.i3 >> new source -> compiling TextSubrTbl.i3 >> new source -> compiling TextSubrTbl.m3 >> new source -> compiling OSTest.m3 >> "../src/OSTest.m3", line 298: incompatible types (i) >> 1 error encountered >> compilation failed => not building program "OSTest" >> performing pathname-tests... >> /home/m3/work/cm3-ws/birch.elegosoft.com-2010-03-01-02-34-08/cm3/m3-libs/libm3/tests/os/AMD64_LINUX >> Fatal Error: package build failed >> >> cm3 returned 2 >> >> Would anybody care to fix that? >> >> Olaf >> -- >> Olaf Wagner -- elego Software Solutions GmbH >> Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany >> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 >> http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin >> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 >> > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 2 14:16:03 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 2 Mar 2010 13:16:03 +0000 Subject: [M3devel] FW: optimizing range checks? In-Reply-To: <20100302125931.619312474001@birch.elegosoft.com> References: <20100302125931.619312474001@birch.elegosoft.com> Message-ID: Hey that's pretty clever. It costs a register, but given: if (b >= constantX|| b <= -constantY) a = 0; The C compiler instead does something like: if ((b + constantY - 1) > (constantX + constantY - 1)) a = 0; This is something the front end could do in many cases.? Adding a constant to eliminate one of the compares and branches is a win. If an x86 compiler will give up a register for this, then it is probably a win everywhere. Granted, it probably requires silent overflow. Oh well. - Jay > Date: Tue, 2 Mar 2010 13:59:31 +0000 > To: m3commit at elegosoft.com > From: jkrell at elego.de > Subject: [M3commit] CVS Update: cm3 > > CVSROOT: /usr/cvs > Changes by: jkrell at birch. 10/03/02 13:59:31 > > Modified files: > cm3/m3-libs/m3core/src/Csupport/Common/: hand.c > > Log message: > remove m3_shift64, which for the record was: > uint64 __stdcall m3_shift64(uint64 a, int b) > { > if (b >= 64 || b <= -64) > a = 0; > else if (b > 0) > a <<= b; > else if (b < 0) > a >>= -b; > return a; > } > > _m3_shift64 at 12: > 00000000: 55 push ebp > 00000001: 8B EC mov ebp,esp > 00000003: 8B 4D 10 mov ecx,dword ptr [ebp+10h] > 00000006: 8D 41 3F lea eax,[ecx+3Fh] > 00000009: 83 F8 7E cmp eax,7Eh > 0000000C: 77 1C ja 0000002A > 0000000E: 8B 45 08 mov eax,dword ptr [ebp+8] > 00000011: 8B 55 0C mov edx,dword ptr [ebp+0Ch] > 00000014: 85 C9 test ecx,ecx > 00000016: 7E 07 jle 0000001F > 00000018: E8 00 00 00 00 call __allshl > 0000001D: EB 0F jmp 0000002E > 0000001F: 7D 0D jge 0000002E > 00000021: F7 D9 neg ecx > 00000023: E8 00 00 00 00 call __aullshr > 00000028: EB 04 jmp 0000002E > 0000002A: 33 C0 xor eax,eax > 0000002C: 33 D2 xor edx,edx > 0000002E: 5D pop ebp > 0000002F: C2 0C 00 ret 0Ch > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 2 15:04:47 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 2 Mar 2010 14:04:47 +0000 Subject: [M3devel] overshifting? Message-ID: PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); I'm guessing the warning is all you're supposed to get here? C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 --- building in NT386 --- new source -> compiling Main.m3 "..\Main.m3", line 894: warning: value out of range *** *** runtime error: *** An enumeration or subrange value was out of range. *** file "..\src\TWordN.m3", line 129 *** Stack trace: FP PC Procedure --------- --------- ------------------------------- 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti nWord\Shift.mg 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 ......... ......... ... more frames ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 19:54:54 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 13:54:54 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: Message-ID: <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. On 2 Mar 2010, at 09:04, Jay K wrote: > PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > > > I'm guessing the warning is all you're supposed to get here? > > > C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > --- building in NT386 --- > new source -> compiling Main.m3 > "..\Main.m3", line 894: warning: value out of range > > *** > *** runtime error: > *** An enumeration or subrange value was out of range. > *** file "..\src\TWordN.m3", line 129 > *** > Stack trace: > FP PC Procedure > --------- --------- ------------------------------- > 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > nWord\Shift.mg > 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > ......... ......... ... more frames ... > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 19:55:38 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 13:55:38 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: Message-ID: <06913ED3-F44E-4A00-822C-8DC1F06E6C06@cs.purdue.edu> PS The warning is there to say that there might be a runtime error. On 2 Mar 2010, at 09:04, Jay K wrote: > PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > > > I'm guessing the warning is all you're supposed to get here? > > > C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > --- building in NT386 --- > new source -> compiling Main.m3 > "..\Main.m3", line 894: warning: value out of range > > *** > *** runtime error: > *** An enumeration or subrange value was out of range. > *** file "..\src\TWordN.m3", line 129 > *** > Stack trace: > FP PC Procedure > --------- --------- ------------------------------- > 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > nWord\Shift.mg > 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > ......... ......... ... more frames ... > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 19:58:05 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 13:58:05 -0500 Subject: [M3devel] FW: optimizing range checks? In-Reply-To: References: <20100302125931.619312474001@birch.elegosoft.com> Message-ID: The front-end is supposed to be machine-independent. It has no idea what the underlying machine will do on overflow. On 2 Mar 2010, at 08:16, Jay K wrote: > Hey that's pretty clever. > It costs a register, but given: > > if (b >= constantX|| b <= -constantY) > a = 0; > > The C compiler instead does something like: > if ((b + constantY - 1) > (constantX + constantY - 1)) > a = 0; > > This is something the front end could do in many cases.? > Adding a constant to eliminate one of the compares and branches is a win. > If an x86 compiler will give up a register for this, then it is probably a win everywhere. > > Granted, it probably requires silent overflow. Oh well. > > - Jay > > > Date: Tue, 2 Mar 2010 13:59:31 +0000 > > To: m3commit at elegosoft.com > > From: jkrell at elego.de > > Subject: [M3commit] CVS Update: cm3 > > > > CVSROOT: /usr/cvs > > Changes by: jkrell at birch. 10/03/02 13:59:31 > > > > Modified files: > > cm3/m3-libs/m3core/src/Csupport/Common/: hand.c > > > > Log message: > > remove m3_shift64, which for the record was: > > uint64 __stdcall m3_shift64(uint64 a, int b) > > { > > if (b >= 64 || b <= -64) > > a = 0; > > else if (b > 0) > > a <<= b; > > else if (b < 0) > > a >>= -b; > > return a; > > } > > > > _m3_shift64 at 12: > > 00000000: 55 push ebp > > 00000001: 8B EC mov ebp,esp > > 00000003: 8B 4D 10 mov ecx,dword ptr [ebp+10h] > > 00000006: 8D 41 3F lea eax,[ecx+3Fh] > > 00000009: 83 F8 7E cmp eax,7Eh > > 0000000C: 77 1C ja 0000002A > > 0000000E: 8B 45 08 mov eax,dword ptr [ebp+8] > > 00000011: 8B 55 0C mov edx,dword ptr [ebp+0Ch] > > 00000014: 85 C9 test ecx,ecx > > 00000016: 7E 07 jle 0000001F > > 00000018: E8 00 00 00 00 call __allshl > > 0000001D: EB 0F jmp 0000002E > > 0000001F: 7D 0D jge 0000002E > > 00000021: F7 D9 neg ecx > > 00000023: E8 00 00 00 00 call __aullshr > > 00000028: EB 04 jmp 0000002E > > 0000002A: 33 C0 xor eax,eax > > 0000002C: 33 D2 xor edx,edx > > 0000002E: 5D pop ebp > > 0000002F: C2 0C 00 ret 0Ch > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 20:50:27 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 14:50:27 -0500 Subject: [M3devel] overshifting? In-Reply-To: <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu> References: <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu> <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu> <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu> Message-ID: <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu> Oh, yes, of course your backend should not blow up on this. On 2 Mar 2010, at 14:46, Tony Hosking wrote: > P.S. Here are the signatures of Shift, RightShift, and LeftShift. > > PROCEDURE Shift (x: T; n: INTEGER): T; > For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n > 0 is like multiplying by 2^(n) > PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > = Shift (x, n) > PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > = Shift (x, -n) > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 2 Mar 2010, at 14:44, Tony Hosking wrote: > >> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. >> >> So, in fact the run-time error is correct! 630 is not in the range [0..63]. >> >> On 2 Mar 2010, at 13:54, Tony Hosking wrote: >> >>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >>> >>> On 2 Mar 2010, at 09:04, Jay K wrote: >>> >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >>>> >>>> >>>> I'm guessing the warning is all you're supposed to get here? >>>> >>>> >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >>>> --- building in NT386 --- >>>> new source -> compiling Main.m3 >>>> "..\Main.m3", line 894: warning: value out of range >>>> >>>> *** >>>> *** runtime error: >>>> *** An enumeration or subrange value was out of range. >>>> *** file "..\src\TWordN.m3", line 129 >>>> *** >>>> Stack trace: >>>> FP PC Procedure >>>> --------- --------- ------------------------------- >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >>>> nWord\Shift.mg >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >>>> ......... ......... ... more frames ... >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 20:44:21 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 14:44:21 -0500 Subject: [M3devel] overshifting? In-Reply-To: <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu> References: <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu> Message-ID: <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. So, in fact the run-time error is correct! 630 is not in the range [0..63]. On 2 Mar 2010, at 13:54, Tony Hosking wrote: > I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > > On 2 Mar 2010, at 09:04, Jay K wrote: > >> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >> >> >> I'm guessing the warning is all you're supposed to get here? >> >> >> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >> --- building in NT386 --- >> new source -> compiling Main.m3 >> "..\Main.m3", line 894: warning: value out of range >> >> *** >> *** runtime error: >> *** An enumeration or subrange value was out of range. >> *** file "..\src\TWordN.m3", line 129 >> *** >> Stack trace: >> FP PC Procedure >> --------- --------- ------------------------------- >> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >> nWord\Shift.mg >> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >> ......... ......... ... more frames ... >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 2 20:46:36 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 14:46:36 -0500 Subject: [M3devel] overshifting? In-Reply-To: <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu> References: <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu> <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu> Message-ID: <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu> P.S. Here are the signatures of Shift, RightShift, and LeftShift. PROCEDURE Shift (x: T; n: INTEGER): T; For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n > 0 is like multiplying by 2^(n) PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; = Shift (x, n) PROCEDURE RightShift (x: T; n: [0..Size-1]): T; = Shift (x, -n) Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 2 Mar 2010, at 14:44, Tony Hosking wrote: > Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > > So, in fact the run-time error is correct! 630 is not in the range [0..63]. > > On 2 Mar 2010, at 13:54, Tony Hosking wrote: > >> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >> >> On 2 Mar 2010, at 09:04, Jay K wrote: >> >>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >>> >>> >>> I'm guessing the warning is all you're supposed to get here? >>> >>> >>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >>> --- building in NT386 --- >>> new source -> compiling Main.m3 >>> "..\Main.m3", line 894: warning: value out of range >>> >>> *** >>> *** runtime error: >>> *** An enumeration or subrange value was out of range. >>> *** file "..\src\TWordN.m3", line 129 >>> *** >>> Stack trace: >>> FP PC Procedure >>> --------- --------- ------------------------------- >>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >>> nWord\Shift.mg >>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >>> ......... ......... ... more frames ... >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 2 22:09:54 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 2 Mar 2010 21:09:54 +0000 Subject: [M3devel] overshifting? In-Reply-To: <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu> References: , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu> Message-ID: I fixed it. It falls back to generating slower code, which is then not hit because it guaranteeably hits a runtime error ahead of it. Perhaps it should just issue a breakpoint there as the code is not reachable. e.g. in C we have: __declspec(noreturn) abort(); void F1() { abort(); /* breakpoint here */ } Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? - Jay From: hosking at cs.purdue.edu Date: Tue, 2 Mar 2010 14:50:27 -0500 To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com; jay.krell at cornell.edu Subject: Re: [M3devel] overshifting? Oh, yes, of course your backend should not blow up on this. On 2 Mar 2010, at 14:46, Tony Hosking wrote: P.S. Here are the signatures of Shift, RightShift, and LeftShift. PROCEDURE Shift (x: T; n: INTEGER): T; For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n > 0 is like multiplying by 2^(n)PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; = Shift (x, n)PROCEDURE RightShift (x: T; n: [0..Size-1]): T; = Shift (x, -n) Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 2 Mar 2010, at 14:44, Tony Hosking wrote: Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. So, in fact the run-time error is correct! 630 is not in the range [0..63]. On 2 Mar 2010, at 13:54, Tony Hosking wrote: I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. On 2 Mar 2010, at 09:04, Jay K wrote: PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); I'm guessing the warning is all you're supposed to get here? C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 --- building in NT386 --- new source -> compiling Main.m3 "..\Main.m3", line 894: warning: value out of range *** *** runtime error: *** An enumeration or subrange value was out of range. *** file "..\src\TWordN.m3", line 129 *** Stack trace: FP PC Procedure --------- --------- ------------------------------- 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti nWord\Shift.mg 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 ......... ......... ... more frames ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 3 00:53:46 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 2 Mar 2010 23:53:46 +0000 Subject: [M3devel] overshifting? In-Reply-To: References: , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, Message-ID: The front end shouldn't bother asking the backend to shift by overlarge constants, eh? Granted, it could be from Shift(a, LAST(INTEGER) + ....); so the frontend can't always know, even if the backend thinks it does. I need to try some things there -- code that produces a large shift via overflowing constants. Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: PROCEDURE Crash() BEGIN EVAL VAL(2, BOOLEAN); END Crash. mostly these are warnings at compile, runtime errors. But some of these things are now barred by m3back, errors, e.g. EVAL LAST(INTEGER) + 1; Maybe I do need that warning callback after all? - Jay ________________________________ > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Tue, 2 Mar 2010 21:09:54 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] overshifting? > > > > > > > > > I fixed it. It falls back to generating slower code, which is then not hit because > > it guaranteeably hits a runtime error ahead of it. > > Perhaps it should just issue a breakpoint there as the code is not reachable. > > e.g. in C we have: > > > > __declspec(noreturn) abort(); > > > > void F1() > > { > abort(); > > /* breakpoint here */ > > } > > > > Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? > > > > - Jay > > > ________________________________ > > From: hosking at cs.purdue.edu > Date: Tue, 2 Mar 2010 14:50:27 -0500 > To: hosking at cs.purdue.edu > CC: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: Re: [M3devel] overshifting? > > > Oh, yes, of course your backend should not blow up on this. > > > > > > > On 2 Mar 2010, at 14:46, Tony Hosking wrote: > > > > P.S. Here are the signatures of Shift, RightShift, and LeftShift. > > > > PROCEDURE Shift (x: T; n: INTEGER): T; > > For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) > > PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > > = Shift (x, n) > > PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > > = Shift (x, -n) > > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue University > > 305 N. University Street | West Lafayette | IN 47907 | USA > > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > > > On 2 Mar 2010, at 14:44, Tony Hosking wrote: > > > > > > Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > > > So, in fact the run-time error is correct! 630 is not in the range [0..63]. > > > > On 2 Mar 2010, at 13:54, Tony Hosking wrote: > > > > > I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > > > > On 2 Mar 2010, at 09:04, Jay K wrote: > > > > PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > > > I'm guessing the warning is all you're supposed to get here? > > > C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > --- building in NT386 --- > new source -> compiling Main.m3 > "..\Main.m3", line 894: warning: value out of range > > *** > *** runtime error: > *** An enumeration or subrange value was out of range. > *** file "..\src\TWordN.m3", line 129 > *** > Stack trace: > FP PC Procedure > --------- --------- ------------------------------- > 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > nWord\Shift.mg > 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > ......... ......... ... more frames ... > > > > > From hosking at cs.purdue.edu Wed Mar 3 04:29:47 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 22:29:47 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, Message-ID: The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. On 2 Mar 2010, at 18:53, Jay K wrote: > > The front end shouldn't bother asking the backend to shift by overlarge constants, eh? > Granted, it could be from > Shift(a, LAST(INTEGER) + ....); > > so the frontend can't always know, even if the backend thinks it does. > I need to try some things there -- code that produces a large shift via overflowing constants. > > Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: > > PROCEDURE Crash() > BEGIN > EVAL VAL(2, BOOLEAN); > END Crash. > > > mostly these are warnings at compile, runtime errors. > But some of these things are now barred by m3back, errors, e.g. > > EVAL LAST(INTEGER) + 1; > > Maybe I do need that warning callback after all? > > > - Jay > > > ________________________________ >> From: jay.krell at cornell.edu >> To: hosking at cs.purdue.edu >> Date: Tue, 2 Mar 2010 21:09:54 +0000 >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] overshifting? >> >> >> >> >> >> >> >> >> I fixed it. It falls back to generating slower code, which is then not hit because >> >> it guaranteeably hits a runtime error ahead of it. >> >> Perhaps it should just issue a breakpoint there as the code is not reachable. >> >> e.g. in C we have: >> >> >> >> __declspec(noreturn) abort(); >> >> >> >> void F1() >> >> { >> abort(); >> >> /* breakpoint here */ >> >> } >> >> >> >> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? >> >> >> >> - Jay >> >> >> ________________________________ >> >> From: hosking at cs.purdue.edu >> Date: Tue, 2 Mar 2010 14:50:27 -0500 >> To: hosking at cs.purdue.edu >> CC: m3devel at elegosoft.com; jay.krell at cornell.edu >> Subject: Re: [M3devel] overshifting? >> >> >> Oh, yes, of course your backend should not blow up on this. >> >> >> >> >> >> >> On 2 Mar 2010, at 14:46, Tony Hosking wrote: >> >> >> >> P.S. Here are the signatures of Shift, RightShift, and LeftShift. >> >> >> >> PROCEDURE Shift (x: T; n: INTEGER): T; >> >> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) >> >> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; >> >> = Shift (x, n) >> >> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; >> >> = Shift (x, -n) >> >> >> >> >> >> Antony Hosking | Associate Professor | Computer Science | Purdue University >> >> 305 N. University Street | West Lafayette | IN 47907 | USA >> >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> >> >> >> On 2 Mar 2010, at 14:44, Tony Hosking wrote: >> >> >> >> >> >> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. >> >> >> So, in fact the run-time error is correct! 630 is not in the range [0..63]. >> >> >> >> On 2 Mar 2010, at 13:54, Tony Hosking wrote: >> >> >> >> >> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >> >> >> >> On 2 Mar 2010, at 09:04, Jay K wrote: >> >> >> >> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >> >> >> I'm guessing the warning is all you're supposed to get here? >> >> >> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >> --- building in NT386 --- >> new source -> compiling Main.m3 >> "..\Main.m3", line 894: warning: value out of range >> >> *** >> *** runtime error: >> *** An enumeration or subrange value was out of range. >> *** file "..\src\TWordN.m3", line 129 >> *** >> Stack trace: >> FP PC Procedure >> --------- --------- ------------------------------- >> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >> nWord\Shift.mg >> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >> ......... ......... ... more frames ... >> >> >> >> >> From jay.krell at cornell.edu Wed Mar 3 05:21:02 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 3 Mar 2010 04:21:02 +0000 Subject: [M3devel] overshifting? In-Reply-To: References: , ,,<9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, ,,<72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, ,,<48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , Message-ID: The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. Doesn't that seem wrong in some way? front end should warn and then backend can be quiet? Why does the backend know more than frontend? Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? - Jay ---------------------------------------- > From: hosking at cs.purdue.edu > Date: Tue, 2 Mar 2010 22:29:47 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] overshifting? > > The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. > > On 2 Mar 2010, at 18:53, Jay K wrote: > >> >> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? >> Granted, it could be from >> Shift(a, LAST(INTEGER) + ....); >> >> so the frontend can't always know, even if the backend thinks it does. >> I need to try some things there -- code that produces a large shift via overflowing constants. >> >> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: >> >> PROCEDURE Crash() >> BEGIN >> EVAL VAL(2, BOOLEAN); >> END Crash. >> >> >> mostly these are warnings at compile, runtime errors. >> But some of these things are now barred by m3back, errors, e.g. >> >> EVAL LAST(INTEGER) + 1; >> >> Maybe I do need that warning callback after all? >> >> >> - Jay >> >> >> ________________________________ >>> From: jay.krell at cornell.edu >>> To: hosking at cs.purdue.edu >>> Date: Tue, 2 Mar 2010 21:09:54 +0000 >>> CC: m3devel at elegosoft.com >>> Subject: Re: [M3devel] overshifting? >>> >>> >>> >>> >>> >>> >>> >>> >>> I fixed it. It falls back to generating slower code, which is then not hit because >>> >>> it guaranteeably hits a runtime error ahead of it. >>> >>> Perhaps it should just issue a breakpoint there as the code is not reachable. >>> >>> e.g. in C we have: >>> >>> >>> >>> __declspec(noreturn) abort(); >>> >>> >>> >>> void F1() >>> >>> { >>> abort(); >>> >>> /* breakpoint here */ >>> >>> } >>> >>> >>> >>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? >>> >>> >>> >>> - Jay >>> >>> >>> ________________________________ >>> >>> From: hosking at cs.purdue.edu >>> Date: Tue, 2 Mar 2010 14:50:27 -0500 >>> To: hosking at cs.purdue.edu >>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu >>> Subject: Re: [M3devel] overshifting? >>> >>> >>> Oh, yes, of course your backend should not blow up on this. >>> >>> >>> >>> >>> >>> >>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: >>> >>> >>> >>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. >>> >>> >>> >>> PROCEDURE Shift (x: T; n: INTEGER): T; >>> >>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) >>> >>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; >>> >>> = Shift (x, n) >>> >>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; >>> >>> = Shift (x, -n) >>> >>> >>> >>> >>> >>> Antony Hosking | Associate Professor | Computer Science | Purdue University >>> >>> 305 N. University Street | West Lafayette | IN 47907 | USA >>> >>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>> >>> >>> >>> >>> >>> >>> >>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: >>> >>> >>> >>> >>> >>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. >>> >>> >>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. >>> >>> >>> >>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: >>> >>> >>> >>> >>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >>> >>> >>> >>> On 2 Mar 2010, at 09:04, Jay K wrote: >>> >>> >>> >>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >>> >>> >>> I'm guessing the warning is all you're supposed to get here? >>> >>> >>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >>> --- building in NT386 --- >>> new source -> compiling Main.m3 >>> "..\Main.m3", line 894: warning: value out of range >>> >>> *** >>> *** runtime error: >>> *** An enumeration or subrange value was out of range. >>> *** file "..\src\TWordN.m3", line 129 >>> *** >>> Stack trace: >>> FP PC Procedure >>> --------- --------- ------------------------------- >>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >>> nWord\Shift.mg >>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >>> ......... ......... ... more frames ... >>> >>> >>> >>> >>> > From jay.krell at cornell.edu Wed Mar 3 05:23:19 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 3 Mar 2010 04:23:19 +0000 Subject: [M3devel] overshifting? In-Reply-To: References: , , , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , , , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , , , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , Message-ID: > You cannot optimise away operations that may cause a run-time error. We don't. We used to. In many cases we absolutely know if there will be overflow or not. If not, optimize. If so, currently, error. But I think maybe it should warn and just call the runtime error functions. Maybe. And I think this is all portable and can be in the frontend. I'm repeating myself now. - Jay ---------------------------------------- > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] overshifting? > Date: Wed, 3 Mar 2010 04:21:02 +0000 > > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > Doesn't that seem wrong in some way? > front end should warn and then backend can be quiet? > Why does the backend know more than frontend? > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > - Jay > > > ---------------------------------------- >> From: hosking at cs.purdue.edu >> Date: Tue, 2 Mar 2010 22:29:47 -0500 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] overshifting? >> >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. >> >> On 2 Mar 2010, at 18:53, Jay K wrote: >> >>> >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? >>> Granted, it could be from >>> Shift(a, LAST(INTEGER) + ....); >>> >>> so the frontend can't always know, even if the backend thinks it does. >>> I need to try some things there -- code that produces a large shift via overflowing constants. >>> >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: >>> >>> PROCEDURE Crash() >>> BEGIN >>> EVAL VAL(2, BOOLEAN); >>> END Crash. >>> >>> >>> mostly these are warnings at compile, runtime errors. >>> But some of these things are now barred by m3back, errors, e.g. >>> >>> EVAL LAST(INTEGER) + 1; >>> >>> Maybe I do need that warning callback after all? >>> >>> >>> - Jay >>> >>> >>> ________________________________ >>>> From: jay.krell at cornell.edu >>>> To: hosking at cs.purdue.edu >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 >>>> CC: m3devel at elegosoft.com >>>> Subject: Re: [M3devel] overshifting? >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> I fixed it. It falls back to generating slower code, which is then not hit because >>>> >>>> it guaranteeably hits a runtime error ahead of it. >>>> >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. >>>> >>>> e.g. in C we have: >>>> >>>> >>>> >>>> __declspec(noreturn) abort(); >>>> >>>> >>>> >>>> void F1() >>>> >>>> { >>>> abort(); >>>> >>>> /* breakpoint here */ >>>> >>>> } >>>> >>>> >>>> >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? >>>> >>>> >>>> >>>> - Jay >>>> >>>> >>>> ________________________________ >>>> >>>> From: hosking at cs.purdue.edu >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 >>>> To: hosking at cs.purdue.edu >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu >>>> Subject: Re: [M3devel] overshifting? >>>> >>>> >>>> Oh, yes, of course your backend should not blow up on this. >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: >>>> >>>> >>>> >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. >>>> >>>> >>>> >>>> PROCEDURE Shift (x: T; n: INTEGER): T; >>>> >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) >>>> >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; >>>> >>>> = Shift (x, n) >>>> >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; >>>> >>>> = Shift (x, -n) >>>> >>>> >>>> >>>> >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University >>>> >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: >>>> >>>> >>>> >>>> >>>> >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. >>>> >>>> >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. >>>> >>>> >>>> >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: >>>> >>>> >>>> >>>> >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >>>> >>>> >>>> >>>> On 2 Mar 2010, at 09:04, Jay K wrote: >>>> >>>> >>>> >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >>>> >>>> >>>> I'm guessing the warning is all you're supposed to get here? >>>> >>>> >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >>>> --- building in NT386 --- >>>> new source -> compiling Main.m3 >>>> "..\Main.m3", line 894: warning: value out of range >>>> >>>> *** >>>> *** runtime error: >>>> *** An enumeration or subrange value was out of range. >>>> *** file "..\src\TWordN.m3", line 129 >>>> *** >>>> Stack trace: >>>> FP PC Procedure >>>> --------- --------- ------------------------------- >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >>>> nWord\Shift.mg >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >>>> ......... ......... ... more frames ... >>>> >>>> >>>> >>>> >>>> >> From hosking at cs.purdue.edu Wed Mar 3 05:44:22 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 2 Mar 2010 23:44:22 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: , , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , Message-ID: <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu> The front-end does fold constants where it can prove no overflow will occur. I don't think I understand any of the rest of what you are saying. On 2 Mar 2010, at 23:21, Jay K wrote: > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > Doesn't that seem wrong in some way? > front end should warn and then backend can be quiet? > Why does the backend know more than frontend? > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > - Jay > > > ---------------------------------------- >> From: hosking at cs.purdue.edu >> Date: Tue, 2 Mar 2010 22:29:47 -0500 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] overshifting? >> >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. >> >> On 2 Mar 2010, at 18:53, Jay K wrote: >> >>> >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? >>> Granted, it could be from >>> Shift(a, LAST(INTEGER) + ....); >>> >>> so the frontend can't always know, even if the backend thinks it does. >>> I need to try some things there -- code that produces a large shift via overflowing constants. >>> >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: >>> >>> PROCEDURE Crash() >>> BEGIN >>> EVAL VAL(2, BOOLEAN); >>> END Crash. >>> >>> >>> mostly these are warnings at compile, runtime errors. >>> But some of these things are now barred by m3back, errors, e.g. >>> >>> EVAL LAST(INTEGER) + 1; >>> >>> Maybe I do need that warning callback after all? >>> >>> >>> - Jay >>> >>> >>> ________________________________ >>>> From: jay.krell at cornell.edu >>>> To: hosking at cs.purdue.edu >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 >>>> CC: m3devel at elegosoft.com >>>> Subject: Re: [M3devel] overshifting? >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> I fixed it. It falls back to generating slower code, which is then not hit because >>>> >>>> it guaranteeably hits a runtime error ahead of it. >>>> >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. >>>> >>>> e.g. in C we have: >>>> >>>> >>>> >>>> __declspec(noreturn) abort(); >>>> >>>> >>>> >>>> void F1() >>>> >>>> { >>>> abort(); >>>> >>>> /* breakpoint here */ >>>> >>>> } >>>> >>>> >>>> >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? >>>> >>>> >>>> >>>> - Jay >>>> >>>> >>>> ________________________________ >>>> >>>> From: hosking at cs.purdue.edu >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 >>>> To: hosking at cs.purdue.edu >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu >>>> Subject: Re: [M3devel] overshifting? >>>> >>>> >>>> Oh, yes, of course your backend should not blow up on this. >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: >>>> >>>> >>>> >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. >>>> >>>> >>>> >>>> PROCEDURE Shift (x: T; n: INTEGER): T; >>>> >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) >>>> >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; >>>> >>>> = Shift (x, n) >>>> >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; >>>> >>>> = Shift (x, -n) >>>> >>>> >>>> >>>> >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University >>>> >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: >>>> >>>> >>>> >>>> >>>> >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. >>>> >>>> >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. >>>> >>>> >>>> >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: >>>> >>>> >>>> >>>> >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. >>>> >>>> >>>> >>>> On 2 Mar 2010, at 09:04, Jay K wrote: >>>> >>>> >>>> >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); >>>> >>>> >>>> I'm guessing the warning is all you're supposed to get here? >>>> >>>> >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 >>>> --- building in NT386 --- >>>> new source -> compiling Main.m3 >>>> "..\Main.m3", line 894: warning: value out of range >>>> >>>> *** >>>> *** runtime error: >>>> *** An enumeration or subrange value was out of range. >>>> *** file "..\src\TWordN.m3", line 129 >>>> *** >>>> Stack trace: >>>> FP PC Procedure >>>> --------- --------- ------------------------------- >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti >>>> nWord\Shift.mg >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 >>>> ......... ......... ... more frames ... >>>> >>>> >>>> >>>> >>>> >> From jay.krell at cornell.edu Wed Mar 3 07:00:46 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 3 Mar 2010 06:00:46 +0000 Subject: [M3devel] overshifting? In-Reply-To: <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu> References: , , ,,<9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , ,,<72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , ,,<48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , ,,<170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , , , , , , <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu> Message-ID: Why does Word.LeftShift(x , 100) even get to the backend? It should just generate the code to issue a runtime error? (even if x could be zero? I think so.) Why doesn't front warn for -FIRST(INTEGER)? It should, right? But still generate code to compute it. Are these just oversights? - Jay > From: hosking at cs.purdue.edu > Date: Tue, 2 Mar 2010 23:44:22 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] overshifting? > > The front-end does fold constants where it can prove no overflow will occur. I don't think I understand any of the rest of what you are saying. > > On 2 Mar 2010, at 23:21, Jay K wrote: > > > > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > > Doesn't that seem wrong in some way? > > front end should warn and then backend can be quiet? > > Why does the backend know more than frontend? > > > > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > > > > > - Jay > > > > > > ---------------------------------------- > >> From: hosking at cs.purdue.edu > >> Date: Tue, 2 Mar 2010 22:29:47 -0500 > >> To: jay.krell at cornell.edu > >> CC: m3devel at elegosoft.com > >> Subject: Re: [M3devel] overshifting? > >> > >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. > >> > >> On 2 Mar 2010, at 18:53, Jay K wrote: > >> > >>> > >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? > >>> Granted, it could be from > >>> Shift(a, LAST(INTEGER) + ....); > >>> > >>> so the frontend can't always know, even if the backend thinks it does. > >>> I need to try some things there -- code that produces a large shift via overflowing constants. > >>> > >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: > >>> > >>> PROCEDURE Crash() > >>> BEGIN > >>> EVAL VAL(2, BOOLEAN); > >>> END Crash. > >>> > >>> > >>> mostly these are warnings at compile, runtime errors. > >>> But some of these things are now barred by m3back, errors, e.g. > >>> > >>> EVAL LAST(INTEGER) + 1; > >>> > >>> Maybe I do need that warning callback after all? > >>> > >>> > >>> - Jay > >>> > >>> > >>> ________________________________ > >>>> From: jay.krell at cornell.edu > >>>> To: hosking at cs.purdue.edu > >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 > >>>> CC: m3devel at elegosoft.com > >>>> Subject: Re: [M3devel] overshifting? > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> I fixed it. It falls back to generating slower code, which is then not hit because > >>>> > >>>> it guaranteeably hits a runtime error ahead of it. > >>>> > >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. > >>>> > >>>> e.g. in C we have: > >>>> > >>>> > >>>> > >>>> __declspec(noreturn) abort(); > >>>> > >>>> > >>>> > >>>> void F1() > >>>> > >>>> { > >>>> abort(); > >>>> > >>>> /* breakpoint here */ > >>>> > >>>> } > >>>> > >>>> > >>>> > >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? > >>>> > >>>> > >>>> > >>>> - Jay > >>>> > >>>> > >>>> ________________________________ > >>>> > >>>> From: hosking at cs.purdue.edu > >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 > >>>> To: hosking at cs.purdue.edu > >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu > >>>> Subject: Re: [M3devel] overshifting? > >>>> > >>>> > >>>> Oh, yes, of course your backend should not blow up on this. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. > >>>> > >>>> > >>>> > >>>> PROCEDURE Shift (x: T; n: INTEGER): T; > >>>> > >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) > >>>> > >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > >>>> > >>>> = Shift (x, n) > >>>> > >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > >>>> > >>>> = Shift (x, -n) > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University > >>>> > >>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>> > >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > >>>> > >>>> > >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> > >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 09:04, Jay K wrote: > >>>> > >>>> > >>>> > >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > >>>> > >>>> > >>>> I'm guessing the warning is all you're supposed to get here? > >>>> > >>>> > >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > >>>> --- building in NT386 --- > >>>> new source -> compiling Main.m3 > >>>> "..\Main.m3", line 894: warning: value out of range > >>>> > >>>> *** > >>>> *** runtime error: > >>>> *** An enumeration or subrange value was out of range. > >>>> *** file "..\src\TWordN.m3", line 129 > >>>> *** > >>>> Stack trace: > >>>> FP PC Procedure > >>>> --------- --------- ------------------------------- > >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > >>>> nWord\Shift.mg > >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > >>>> ......... ......... ... more frames ... > >>>> > >>>> > >>>> > >>>> > >>>> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 3 16:54:00 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 3 Mar 2010 10:54:00 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: , , , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , , , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , , , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , , , , , , <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu> Message-ID: On 3 Mar 2010, at 01:00, Jay K wrote: > Why does Word.LeftShift(x , 100) even get to the backend? It should just generate the code to issue a runtime error? (even if x could be zero? I think so.) Because the language spec says that a range error (100 does not fit in the range [0..Word.Size-1]) is a run-time error. Note that the code generator's M3CG_Ops don't specify ranges. The range check is in the *interface* for Word. The m3middle and backend M3CG_Ops are intended to be more general. So, M3CG_Ops.shift_left and shift_right are permitted to take an argument that is not range-checked. > Why doesn't front warn for -FIRST(INTEGER)? It should, right? This is independent of the range check above. This is an expression that may or may not evaluate depending on the run-time defaults (checking for overflow or not). The compiler must be conservative and let the run-time influence whether overflow checking is in place or not. Overflow checking is a run-time option. The compiler must defer to the run-time. > But still generate code to compute it. > Are these just oversights? Nope. Intentional. > > - Jay > > > From: hosking at cs.purdue.edu > > Date: Tue, 2 Mar 2010 23:44:22 -0500 > > To: jay.krell at cornell.edu > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] overshifting? > > > > The front-end does fold constants where it can prove no overflow will occur. I don't think I understand any of the rest of what you are saying. > > > > On 2 Mar 2010, at 23:21, Jay K wrote: > > > > > > > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > > > > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > > > > > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > > > > > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > > > Doesn't that seem wrong in some way? > > > front end should warn and then backend can be quiet? > > > Why does the backend know more than frontend? > > > > > > > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > > > > > > > > > - Jay > > > > > > > > > ---------------------------------------- > > >> From: hosking at cs.purdue.edu > > >> Date: Tue, 2 Mar 2010 22:29:47 -0500 > > >> To: jay.krell at cornell.edu > > >> CC: m3devel at elegosoft.com > > >> Subject: Re: [M3devel] overshifting? > > >> > > >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. > > >> > > >> On 2 Mar 2010, at 18:53, Jay K wrote: > > >> > > >>> > > >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? > > >>> Granted, it could be from > > >>> Shift(a, LAST(INTEGER) + ....); > > >>> > > >>> so the frontend can't always know, even if the backend thinks it does. > > >>> I need to try some things there -- code that produces a large shift via overflowing constants. > > >>> > > >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: > > >>> > > >>> PROCEDURE Crash() > > >>> BEGIN > > >>> EVAL VAL(2, BOOLEAN); > > >>> END Crash. > > >>> > > >>> > > >>> mostly these are warnings at compile, runtime errors. > > >>> But some of these things are now barred by m3back, errors, e.g. > > >>> > > >>> EVAL LAST(INTEGER) + 1; > > >>> > > >>> Maybe I do need that warning callback after all? > > >>> > > >>> > > >>> - Jay > > >>> > > >>> > > >>> ________________________________ > > >>>> From: jay.krell at cornell.edu > > >>>> To: hosking at cs.purdue.edu > > >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 > > >>>> CC: m3devel at elegosoft.com > > >>>> Subject: Re: [M3devel] overshifting? > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> I fixed it. It falls back to generating slower code, which is then not hit because > > >>>> > > >>>> it guaranteeably hits a runtime error ahead of it. > > >>>> > > >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. > > >>>> > > >>>> e.g. in C we have: > > >>>> > > >>>> > > >>>> > > >>>> __declspec(noreturn) abort(); > > >>>> > > >>>> > > >>>> > > >>>> void F1() > > >>>> > > >>>> { > > >>>> abort(); > > >>>> > > >>>> /* breakpoint here */ > > >>>> > > >>>> } > > >>>> > > >>>> > > >>>> > > >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? > > >>>> > > >>>> > > >>>> > > >>>> - Jay > > >>>> > > >>>> > > >>>> ________________________________ > > >>>> > > >>>> From: hosking at cs.purdue.edu > > >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 > > >>>> To: hosking at cs.purdue.edu > > >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu > > >>>> Subject: Re: [M3devel] overshifting? > > >>>> > > >>>> > > >>>> Oh, yes, of course your backend should not blow up on this. > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. > > >>>> > > >>>> > > >>>> > > >>>> PROCEDURE Shift (x: T; n: INTEGER): T; > > >>>> > > >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) > > >>>> > > >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > > >>>> > > >>>> = Shift (x, n) > > >>>> > > >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > > >>>> > > >>>> = Shift (x, -n) > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University > > >>>> > > >>>> 305 N. University Street | West Lafayette | IN 47907 | USA > > >>>> > > >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > > >>>> > > >>>> > > >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 09:04, Jay K wrote: > > >>>> > > >>>> > > >>>> > > >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > > >>>> > > >>>> > > >>>> I'm guessing the warning is all you're supposed to get here? > > >>>> > > >>>> > > >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > > >>>> --- building in NT386 --- > > >>>> new source -> compiling Main.m3 > > >>>> "..\Main.m3", line 894: warning: value out of range > > >>>> > > >>>> *** > > >>>> *** runtime error: > > >>>> *** An enumeration or subrange value was out of range. > > >>>> *** file "..\src\TWordN.m3", line 129 > > >>>> *** > > >>>> Stack trace: > > >>>> FP PC Procedure > > >>>> --------- --------- ------------------------------- > > >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > > >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > > >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > > >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > > >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > > >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > > >>>> nWord\Shift.mg > > >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > > >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > > >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > > >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > > >>>> ......... ......... ... more frames ... > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 4 00:52:54 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 3 Mar 2010 23:52:54 +0000 Subject: [M3devel] overshifting? In-Reply-To: References: , , , ,,<9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , , ,,<72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , , ,,<48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , ,,<170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , ,,, ,,, , , , , , , <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu>, , Message-ID: Tony you aren't understanding me. I'm not suggesting to optimize away runtime errors. I understand that part of the spec. It has been referenced multiple times recently. I'm saying that the front end knows often/always without a doubt if an operation will overflow or not. If the operation will not overflow, it can do the work itself. From what you say, it already does. (I'm going to confuse "overflow" and "subrange error" here briefly.) If the operation will overflow, it can just generate code to issue the error. Now, for most operations, it doesn't know how the target responds to overflow, so the second statement isn't quite right. However, if the operation absolutely will overflow, the frontend should warn. It at least misses some cases, e.g. -FIRST(INTEGER). Granted, on a one's complement system, it won't. But it should warn for the sake of all two's complement systems, which is all of them. Let's amend the previous: If the operation absolutely will overflow, the front end should warn, and then just ask CG to generate the usual code, as if the operation wouldn't overflow. I think it misses some warnings currently. I think it's easy to fix and I'll look into it. Basically whenever the TInt operations fail, should warn. Now, let's somewhat replace "overflow" with "subrange error". LeftShift(x, 100) acts the same on all targets. It isn't overflow, it is subrange error. In this case, I assert, the front end should warn, and it shouldn't bother asking the backend to generate code to the shift. -Jay From: hosking at cs.purdue.edu Date: Wed, 3 Mar 2010 10:54:00 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] overshifting? On 3 Mar 2010, at 01:00, Jay K wrote: Why does Word.LeftShift(x , 100) even get to the backend? It should just generate the code to issue a runtime error? (even if x could be zero? I think so.) Because the language spec says that a range error (100 does not fit in the range [0..Word.Size-1]) is a run-time error. Note that the code generator's M3CG_Ops don't specify ranges. The range check is in the *interface* for Word. The m3middle and backend M3CG_Ops are intended to be more general. So, M3CG_Ops.shift_left and shift_right are permitted to take an argument that is not range-checked. Why doesn't front warn for -FIRST(INTEGER)? It should, right? This is independent of the range check above. This is an expression that may or may not evaluate depending on the run-time defaults (checking for overflow or not). The compiler must be conservative and let the run-time influence whether overflow checking is in place or not. Overflow checking is a run-time option. The compiler must defer to the run-time. But still generate code to compute it. Are these just oversights? Nope. Intentional. - Jay > From: hosking at cs.purdue.edu > Date: Tue, 2 Mar 2010 23:44:22 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] overshifting? > > The front-end does fold constants where it can prove no overflow will occur. I don't think I understand any of the rest of what you are saying. > > On 2 Mar 2010, at 23:21, Jay K wrote: > > > > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > > Doesn't that seem wrong in some way? > > front end should warn and then backend can be quiet? > > Why does the backend know more than frontend? > > > > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > > > > > - Jay > > > > > > ---------------------------------------- > >> From: hosking at cs.purdue.edu > >> Date: Tue, 2 Mar 2010 22:29:47 -0500 > >> To: jay.krell at cornell.edu > >> CC: m3devel at elegosoft.com > >> Subject: Re: [M3devel] overshifting? > >> > >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. > >> > >> On 2 Mar 2010, at 18:53, Jay K wrote: > >> > >>> > >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? > >>> Granted, it could be from > >>> Shift(a, LAST(INTEGER) + ....); > >>> > >>> so the frontend can't always know, even if the backend thinks it does. > >>> I need to try some things there -- code that produces a large shift via overflowing constants. > >>> > >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: > >>> > >>> PROCEDURE Crash() > >>> BEGIN > >>> EVAL VAL(2, BOOLEAN); > >>> END Crash. > >>> > >>> > >>> mostly these are warnings at compile, runtime errors. > >>> But some of these things are now barred by m3back, errors, e.g. > >>> > >>> EVAL LAST(INTEGER) + 1; > >>> > >>> Maybe I do need that warning callback after all? > >>> > >>> > >>> - Jay > >>> > >>> > >>> ________________________________ > >>>> From: jay.krell at cornell.edu > >>>> To: hosking at cs.purdue.edu > >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 > >>>> CC: m3devel at elegosoft.com > >>>> Subject: Re: [M3devel] overshifting? > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> I fixed it. It falls back to generating slower code, which is then not hit because > >>>> > >>>> it guaranteeably hits a runtime error ahead of it. > >>>> > >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. > >>>> > >>>> e.g. in C we have: > >>>> > >>>> > >>>> > >>>> __declspec(noreturn) abort(); > >>>> > >>>> > >>>> > >>>> void F1() > >>>> > >>>> { > >>>> abort(); > >>>> > >>>> /* breakpoint here */ > >>>> > >>>> } > >>>> > >>>> > >>>> > >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? > >>>> > >>>> > >>>> > >>>> - Jay > >>>> > >>>> > >>>> ________________________________ > >>>> > >>>> From: hosking at cs.purdue.edu > >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 > >>>> To: hosking at cs.purdue.edu > >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu > >>>> Subject: Re: [M3devel] overshifting? > >>>> > >>>> > >>>> Oh, yes, of course your backend should not blow up on this. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. > >>>> > >>>> > >>>> > >>>> PROCEDURE Shift (x: T; n: INTEGER): T; > >>>> > >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) > >>>> > >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > >>>> > >>>> = Shift (x, n) > >>>> > >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > >>>> > >>>> = Shift (x, -n) > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University > >>>> > >>>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>>> > >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > >>>> > >>>> > >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: > >>>> > >>>> > >>>> > >>>> > >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > >>>> > >>>> > >>>> > >>>> On 2 Mar 2010, at 09:04, Jay K wrote: > >>>> > >>>> > >>>> > >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > >>>> > >>>> > >>>> I'm guessing the warning is all you're supposed to get here? > >>>> > >>>> > >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > >>>> --- building in NT386 --- > >>>> new source -> compiling Main.m3 > >>>> "..\Main.m3", line 894: warning: value out of range > >>>> > >>>> *** > >>>> *** runtime error: > >>>> *** An enumeration or subrange value was out of range. > >>>> *** file "..\src\TWordN.m3", line 129 > >>>> *** > >>>> Stack trace: > >>>> FP PC Procedure > >>>> --------- --------- ------------------------------- > >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > >>>> nWord\Shift.mg > >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > >>>> ......... ......... ... more frames ... > >>>> > >>>> > >>>> > >>>> > >>>> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 4 01:45:07 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 4 Mar 2010 00:45:07 +0000 Subject: [M3devel] set operations parse.c? Message-ID: The flag to setop is maybe yucky, but I think static void m3cg_set_eq (void) { m3cg_set_eq_or_ne(EQ_EXPR); } static void m3cg_set_ne (void) { m3cg_set_eq_or_ne(NE_EXPR); } is still good. There are two functions *very* similar in functionality that vary in only one token. The counterpoint would be, perhaps, if one is proficient enough in maintaining parse.c, that the two functions are small and one could write them up in a flash without referring to examples. One needs that level of proficiency to make more significant changes anyway. (I'll be removing the set_singleton/member functions shortly, assuming reasonable codegen.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 4 04:20:07 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 3 Mar 2010 22:20:07 -0500 Subject: [M3devel] overshifting? In-Reply-To: References: , , , , , <9D4A85F3-A55B-41F0-856C-A6873069D2BA@cs.purdue.edu>, , , , , <72E1301F-E6A1-4015-A4D2-42CCD9078B9A@cs.purdue.edu>, , , , , <48BEC38A-0881-40FA-92DA-476CEAF49D71@cs.purdue.edu>, , , , , <170A8E16-872C-4BFC-8DEA-213F4F2D1901@cs.purdue.edu>, , , , , , , , , , , , , , <34AFBB3A-2CFA-4175-837F-6578B00AA49C@cs.purdue.edu>, , Message-ID: <4671EB9C-6182-462C-B04F-89428155D530@cs.purdue.edu> On 3 Mar 2010, at 18:52, Jay K wrote: > Tony you aren't understanding me. > > > I'm not suggesting to optimize away runtime errors. > I understand that part of the spec. It has been referenced multiple times recently. > > > I'm saying that the front end knows often/always without a doubt if an operation will overflow or not. > If the operation will not overflow, it can do the work itself. > From what you say, it already does. Correct. > (I'm going to confuse "overflow" and "subrange error" here briefly.) But they are distinct concepts! They cannot be merged. > If the operation will overflow, it can just generate code to issue the error. No! Overflow is a run-time error only if enabled at run-time using FloatMode (or if the target checks for overflow). I think you are not understanding me! > Now, for most operations, it doesn't know how the target responds to overflow, so the second statement isn't quite right. > However, if the operation absolutely will overflow, the frontend should warn. It at least misses some cases, e.g. -FIRST(INTEGER). Granted, on a one's complement system, it won't. But it should warn for the sake of all two's complement systems, which is all of them. > > Let's amend the previous: > If the operation absolutely will overflow, the front end should warn, and then just ask CG to generate the usual code, as if the operation wouldn't overflow. I think it misses some warnings currently. I think it's easy to fix and I'll look into it. Basically whenever the TInt operations fail, should warn. > > > Now, let's somewhat replace "overflow" with "subrange error". > LeftShift(x, 100) > acts the same on all targets. It isn't overflow, it is subrange error. > In this case, I assert, the front end should warn, and it shouldn't bother asking the backend to generate code to the shift. > > > -Jay > > > From: hosking at cs.purdue.edu > Date: Wed, 3 Mar 2010 10:54:00 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] overshifting? > > On 3 Mar 2010, at 01:00, Jay K wrote: > > Why does Word.LeftShift(x , 100) even get to the backend? It should just generate the code to issue a runtime error? (even if x could be zero? I think so.) > > Because the language spec says that a range error (100 does not fit in the range [0..Word.Size-1]) is a run-time error. Note that the code generator's M3CG_Ops don't specify ranges. The range check is in the *interface* for Word. The m3middle and backend M3CG_Ops are intended to be more general. So, M3CG_Ops.shift_left and shift_right are permitted to take an argument that is not range-checked. > > Why doesn't front warn for -FIRST(INTEGER)? It should, right? > > This is independent of the range check above. This is an expression that may or may not evaluate depending on the run-time defaults (checking for overflow or not). The compiler must be conservative and let the run-time influence whether overflow checking is in place or not. > > Overflow checking is a run-time option. The compiler must defer to the run-time. > > But still generate code to compute it. > Are these just oversights? > > Nope. Intentional. > > > - Jay > > > From: hosking at cs.purdue.edu > > Date: Tue, 2 Mar 2010 23:44:22 -0500 > > To: jay.krell at cornell.edu > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] overshifting? > > > > The front-end does fold constants where it can prove no overflow will occur. I don't think I understand any of the rest of what you are saying. > > > > On 2 Mar 2010, at 23:21, Jay K wrote: > > > > > > > > The backend can know if there can be a runtime error -- e.g. if the particular target never checks for overflow. Though that could/should also be exposed in Target.i3? > > > > > > I'm slightly leary of this, because currently the absolute norm is no overflow checking, so it seems promising to provide extra value in that case. But maybe the norm will shift and the value will decrease. That is, Target.i3 could advertise that a target never checks for overflow, and then m3front could fold lots of constants for that target. And if Target.i3 indicates a target might/always check for overflow at runtime, then m3front shouldn't fold them. If hypothecal target always checks for overflow, then m3front could omit the code to do the math and just call the runtime error code. > > > > > > > > > m3back historically has always folded constants, and "never" checked for overflow (ok, the LeftShift/RightShift shift count probably). > > > > > > > > > Only with the 64bit longint has overflow started to be detected by m3back, and it has caught some real cases, where the frontend is quiet. > > > Doesn't that seem wrong in some way? > > > front end should warn and then backend can be quiet? > > > Why does the backend know more than frontend? > > > > > > > > > Part of what I mean here is that even if frontend can't always fold the constants, it can know in many cases that overflow absolutely will not occur, and fold those. Like, do the math at compile time, if no overflow, generate the more optimal code. If overflow, generate the slower code and warn? > > > > > > > > > > > > - Jay > > > > > > > > > ---------------------------------------- > > >> From: hosking at cs.purdue.edu > > >> Date: Tue, 2 Mar 2010 22:29:47 -0500 > > >> To: jay.krell at cornell.edu > > >> CC: m3devel at elegosoft.com > > >> Subject: Re: [M3devel] overshifting? > > >> > > >> The backend should be silent. The warnings should come from the front-end, which has the type information needed. You cannot optimise away operations that may cause a run-time error. > > >> > > >> On 2 Mar 2010, at 18:53, Jay K wrote: > > >> > > >>> > > >>> The front end shouldn't bother asking the backend to shift by overlarge constants, eh? > > >>> Granted, it could be from > > >>> Shift(a, LAST(INTEGER) + ....); > > >>> > > >>> so the frontend can't always know, even if the backend thinks it does. > > >>> I need to try some things there -- code that produces a large shift via overflowing constants. > > >>> > > >>> Which..hm..begets the question. I have the backend error for constant overflows. But shouldn't it actually warn? You know, people code these things to trigger the runtime behavior: > > >>> > > >>> PROCEDURE Crash() > > >>> BEGIN > > >>> EVAL VAL(2, BOOLEAN); > > >>> END Crash. > > >>> > > >>> > > >>> mostly these are warnings at compile, runtime errors. > > >>> But some of these things are now barred by m3back, errors, e.g. > > >>> > > >>> EVAL LAST(INTEGER) + 1; > > >>> > > >>> Maybe I do need that warning callback after all? > > >>> > > >>> > > >>> - Jay > > >>> > > >>> > > >>> ________________________________ > > >>>> From: jay.krell at cornell.edu > > >>>> To: hosking at cs.purdue.edu > > >>>> Date: Tue, 2 Mar 2010 21:09:54 +0000 > > >>>> CC: m3devel at elegosoft.com > > >>>> Subject: Re: [M3devel] overshifting? > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> I fixed it. It falls back to generating slower code, which is then not hit because > > >>>> > > >>>> it guaranteeably hits a runtime error ahead of it. > > >>>> > > >>>> Perhaps it should just issue a breakpoint there as the code is not reachable. > > >>>> > > >>>> e.g. in C we have: > > >>>> > > >>>> > > >>>> > > >>>> __declspec(noreturn) abort(); > > >>>> > > >>>> > > >>>> > > >>>> void F1() > > >>>> > > >>>> { > > >>>> abort(); > > >>>> > > >>>> /* breakpoint here */ > > >>>> > > >>>> } > > >>>> > > >>>> > > >>>> > > >>>> Given the guaranteed runtime error, why does the frontend bother asking the backend to do the shift? > > >>>> > > >>>> > > >>>> > > >>>> - Jay > > >>>> > > >>>> > > >>>> ________________________________ > > >>>> > > >>>> From: hosking at cs.purdue.edu > > >>>> Date: Tue, 2 Mar 2010 14:50:27 -0500 > > >>>> To: hosking at cs.purdue.edu > > >>>> CC: m3devel at elegosoft.com; jay.krell at cornell.edu > > >>>> Subject: Re: [M3devel] overshifting? > > >>>> > > >>>> > > >>>> Oh, yes, of course your backend should not blow up on this. > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 14:46, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> P.S. Here are the signatures of Shift, RightShift, and LeftShift. > > >>>> > > >>>> > > >>>> > > >>>> PROCEDURE Shift (x: T; n: INTEGER): T; > > >>>> > > >>>> For all i such that both i and i - n are in the range [0 .. Word.Size - 1], bit i of the result equals bit i - n of x. The other bits of the result are 0. Thus, shifting by n> 0 is like multiplying by 2^(n) > > >>>> > > >>>> PROCEDURE LeftShift (x: T; n: [0..Size-1]): T; > > >>>> > > >>>> = Shift (x, n) > > >>>> > > >>>> PROCEDURE RightShift (x: T; n: [0..Size-1]): T; > > >>>> > > >>>> = Shift (x, -n) > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> Antony Hosking | Associate Professor | Computer Science | Purdue University > > >>>> > > >>>> 305 N. University Street | West Lafayette | IN 47907 | USA > > >>>> > > >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 14:44, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> Actually, I should have noticed. Word.LeftShift and Word.RightShift both check the range of the shift parameter. It's not the shift that is out of range. It is the shift constant (in this case 630). If you try Shift(FIRST(LONGINT), -630) you get 0 as expected. > > >>>> > > >>>> > > >>>> So, in fact the run-time error is correct! 630 is not in the range [0..63]. > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 13:54, Tony Hosking wrote: > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> I'm trying to understand this. Surely shifting right will never be out of range? Looks like something is broken. > > >>>> > > >>>> > > >>>> > > >>>> On 2 Mar 2010, at 09:04, Jay K wrote: > > >>>> > > >>>> > > >>>> > > >>>> PutLH(Long.RightShift(FIRST(LONGINT), 630)); PutT("\n"); > > >>>> > > >>>> > > >>>> I'm guessing the warning is all you're supposed to get here? > > >>>> > > >>>> > > >>>> C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227>cm3 > > >>>> --- building in NT386 --- > > >>>> new source -> compiling Main.m3 > > >>>> "..\Main.m3", line 894: warning: value out of range > > >>>> > > >>>> *** > > >>>> *** runtime error: > > >>>> *** An enumeration or subrange value was out of range. > > >>>> *** file "..\src\TWordN.m3", line 129 > > >>>> *** > > >>>> Stack trace: > > >>>> FP PC Procedure > > >>>> --------- --------- ------------------------------- > > >>>> 0x12f0dc 0x48791a RightShift + 0x35 in ..\src\TWordN.m3 > > >>>> 0x12f15c 0x47c62f doshift + 0x2c6 in ..\src\Stackx86.m3 > > >>>> 0x12f188 0x4485d2 shift_right + 0x1b2 in ..\src\M3x86.m3 > > >>>> 0x12f1ac 0x6414ed shift_right + 0xf5 in ..\src\M3CG_Check.m3 > > >>>> 0x12f1d4 0x505a4f Shift_right + 0x87 in ..\src\misc\CG.m3 > > >>>> 0x12f1f8 0x570641 CompileR + 0x1da in ..\NT386\LongShift.m3 => ..\src\builti > > >>>> nWord\Shift.mg > > >>>> 0x12f218 0x5cec2f Compile + 0x83 in ..\src\exprs\CallExpr.m3 > > >>>> 0x12f234 0x5bfee4 Compile + 0x53 in ..\src\exprs\Expr.m3 > > >>>> 0x12f274 0x5d19a7 EmitChecks + 0xdf in ..\src\exprs\CheckExpr.m3 > > >>>> 0x12f2b4 0x5c8b15 GenOrdinal + 0xa6 in ..\src\values\Formal.m3 > > >>>> ......... ......... ... more frames ... > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 4 04:27:55 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 3 Mar 2010 22:27:55 -0500 Subject: [M3devel] set operations parse.c? In-Reply-To: References: Message-ID: It obscures the simple intent of the eq/ne operations, and conflates them with other similarly simple set operations. Agreed, the implementations now vary in only one token, but I can read them each in a single page of my editor and validate their correctness without having to trace through multiple calls and track a flag value in context along the way. On 3 Mar 2010, at 19:45, Jay K wrote: > The flag to setop is maybe yucky, but I think > static void m3cg_set_eq (void) { m3cg_set_eq_or_ne(EQ_EXPR); } > static void m3cg_set_ne (void) { m3cg_set_eq_or_ne(NE_EXPR); } > > > is still good. > There are two functions *very* similar in functionality that vary in only one token. > > > The counterpoint would be, perhaps, if one is proficient enough > in maintaining parse.c, that the two functions are small and > one could write them up in a flash without referring to examples. EXACTLY! > One needs that level of proficiency to make more significant changes anyway. > (I'll be removing the set_singleton/member functions shortly, assuming > reasonable codegen.) > > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 4 07:44:41 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 4 Mar 2010 06:44:41 +0000 Subject: [M3devel] overshift/overflow Message-ID: Tony, these two questions seemed to go unanswered: 1) What should this code do: Word.LeftShift(..., 100); (* where 100 >= BITSIZE(INTEGER), and similar for Long.LeftShift and shift >= BITSIZE(LONGINT) *) I think m3front should generate a warning and then either generate the same code as today, or something smaller where it only generates code to issue the runtime error. The warning is missing. "Generating the code same as today" is harder on backends, since they have to check for this condition and handle it somehow, though it doesn't matter much what they do, since the runtime error generation will always run and anything after it never will. 2) What should this code do: EVAL -FIRST(INTEGER); I believe the frontend should issue a warning. And then generate the same code as today. Just the warning is missing. 3) Aside, and not a question. I believe, esp. at this point, the notion that overflow checking is a per-thread setting is a mistake. It has "never" been used, save on a small number of targets. It is too late to foist that change on any code thread-wide. The correct thing to do is introduce different interfaces/modules/types/functions which either always do overflow checking, or, perhaps but less likely, new interfaces/modules/types/functions that are runtime configurable, as INTEGER was originally speced. Even the original spec not so great in my opinion. There are too many things to code correctly, let alone worrying about "this" possibly varying underneath you. The overall system is too much a combination of different code to expect just because one set of code wants integer overflow to be an error, that all the code can deal with that. Better to have separate interfaces/functions/types for the different functionality. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 4 07:55:14 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 4 Mar 2010 06:55:14 +0000 Subject: [M3devel] set operations parse.c? In-Reply-To: References: , Message-ID: Any reuse is too obscuring and not worthwhile if the functions already fit on a page? Huh. By that metric, setop, setop2, m3cg_set_compare aren't needed either. Actually I do find setop vs. setop2 a *little* hard to read. And I hope to soon get rid of two out of its three uses pretty soon (set_singleton, member), maybe tonight. The relationship between eq and ne seems among the strongest, except for all the other comparisons. Actually I'm not sure I "like" a lot here. There is or can be enough similarity in the backends, that the frontend probably should just generate either inline code or calls to portable Modula-3 functions, at least lt/le/gt/ge, and to memcmp for eq/ne. singleton/member I'm a bit on the fence. They can be generated portably inline with pretty small code, though that'd take away the ability of m3back to easily know to use bt/bts instructions. You know, my big "fight" is with "too much code". Anything that the frontend can do pretty darn well, is better than doing it in multiple backends. "More higher level operations" are most useful so a "simpler" backend, such as m3back but not gcc, can more easily generate better code. - Jay Subject: Re: set operations parse.c? From: hosking at cs.purdue.edu Date: Wed, 3 Mar 2010 22:27:55 -0500 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu It obscures the simple intent of the eq/ne operations, and conflates them with other similarly simple set operations. Agreed, the implementations now vary in only one token, but I can read them each in a single page of my editor and validate their correctness without having to trace through multiple calls and track a flag value in context along the way. On 3 Mar 2010, at 19:45, Jay K wrote: The flag to setop is maybe yucky, but I think static void m3cg_set_eq (void) { m3cg_set_eq_or_ne(EQ_EXPR); } static void m3cg_set_ne (void) { m3cg_set_eq_or_ne(NE_EXPR); } is still good. There are two functions *very* similar in functionality that vary in only one token. The counterpoint would be, perhaps, if one is proficient enough in maintaining parse.c, that the two functions are small and one could write them up in a flash without referring to examples. EXACTLY! One needs that level of proficiency to make more significant changes anyway. (I'll be removing the set_singleton/member functions shortly, assuming reasonable codegen.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 4 08:43:31 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 4 Mar 2010 02:43:31 -0500 Subject: [M3devel] set operations parse.c? In-Reply-To: References: , Message-ID: On 4 Mar 2010, at 01:55, Jay K wrote: > Any reuse is too obscuring and not worthwhile if the functions already fit on a page? > Huh. Why hide simple code behind a contorted flag-driven procedure abstraction? > By that metric, setop, setop2, m3cg_set_compare aren't needed either. They don't require calling context to understand what they mean. > Actually I do find setop vs. setop2 a *little* hard to read. ;-) > And I hope to soon get rid of two out of its three uses pretty soon (set_singleton, member), maybe tonight. > > > The relationship between eq and ne seems among the strongest, except for all the other comparisons. > > > Actually I'm not sure I "like" a lot here. > There is or can be enough similarity in the backends, that the frontend probably should just generate either inline code or calls to portable Modula-3 functions, at least lt/le/gt/ge, and to memcmp for eq/ne. Too much work for the front-end. It shouldn't have to worry about such things. > singleton/member I'm a bit on the fence. > They can be generated portably inline with pretty small code, though that'd take away the ability of m3back to easily know to use bt/bts instructions. > > > You know, my big "fight" is with "too much code". Anything that the frontend can do pretty darn well, is better than doing it in multiple backends. "More higher level operations" are most useful so a "simpler" backend, such as m3back but not gcc, can more easily generate better code. > > > - Jay > > > Subject: Re: set operations parse.c? > From: hosking at cs.purdue.edu > Date: Wed, 3 Mar 2010 22:27:55 -0500 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > > It obscures the simple intent of the eq/ne operations, and conflates them with other similarly simple set operations. > > Agreed, the implementations now vary in only one token, but I can read them each in a single page of my editor and validate their correctness without having to trace through multiple calls and track a flag value in context along the way. > > On 3 Mar 2010, at 19:45, Jay K wrote: > > The flag to setop is maybe yucky, but I think > static void m3cg_set_eq (void) { m3cg_set_eq_or_ne(EQ_EXPR); } > static void m3cg_set_ne (void) { m3cg_set_eq_or_ne(NE_EXPR); } > > > is still good. > There are two functions *very* similar in functionality that vary in only one token. > > > The counterpoint would be, perhaps, if one is proficient enough > in maintaining parse.c, that the two functions are small and > one could write them up in a flash without referring to examples. > > EXACTLY! > > One needs that level of proficiency to make more significant changes anyway. > (I'll be removing the set_singleton/member functions shortly, assuming > reasonable codegen.) > > > - Jay > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 4 08:57:26 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 4 Mar 2010 02:57:26 -0500 Subject: [M3devel] overshift/overflow In-Reply-To: References: Message-ID: I already answered these questions. On 4 Mar 2010, at 01:44, Jay K wrote: > Tony, these two questions seemed to go unanswered: > > 1) > > What should this code do: > > > Word.LeftShift(..., 100); (* where 100 >= BITSIZE(INTEGER), > and similar for Long.LeftShift and shift >= BITSIZE(LONGINT) *) > > > I think m3front should generate a warning It does generate a warning. > and then either generate the same code as today, > or something smaller where it only generates code > to issue the runtime error. The code for the shift doesn't know that the warning has been generated, since it is all in the checking of the arguments. > The warning is missing. No, it is generated. > "Generating the code same as today" is harder on backends, > since they have to check for this condition and handle it somehow, > though it doesn't matter much what they do, since the runtime > error generation will always run and anything after it never will. Why can't a backend simply generate code for the large shift, as if it had been a call to Word.Shift(..., 100)? > 2) What should this code do: > > > EVAL -FIRST(INTEGER); > > > I believe the frontend should issue a warning. Why? The front-end does not reason about overflow (except when computing compile-time constants). Overflow is a run-time concept!!!!!!!!!!!!! > And then generate the same code as today. > Just the warning is missing. > > > > 3) Aside, and not a question. > I believe, esp. at this point, the notion that overflow checking is a per-thread setting > is a mistake. It has "never" been used, save on a small number of targets. > It is too late to foist that change on any code thread-wide. > The correct thing to do is introduce different interfaces/modules/types/functions > which either always do overflow checking, or, perhaps but less likely, > new interfaces/modules/types/functions that are runtime configurable, as > INTEGER was originally speced. NOOOOO!!!!!!! That will impose an undue expense on targets where such checking is expensive. Targets where trap handlers can be used to catch overflow will enable it to be turned on via FloatMode. Targets that don't support it efficiently don't have to! > Even the original spec not so great in my opinion. > There are too many things to code correctly, let alone worrying about "this" > possibly varying underneath you. The overall system is too much a combination > of different code to expect just because one set of code wants integer overflow > to be an error, that all the code can deal with that. > Better to have separate interfaces/functions/types for the different functionality. > > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 4 10:00:11 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 4 Mar 2010 09:00:11 +0000 Subject: [M3devel] overshift/overflow In-Reply-To: References: , Message-ID: > Word.LeftShift(..., 100); (* where 100 >= BITSIZE(INTEGER), My mistake, it does generate a warning. It also seems to not generate the code to do the shift, good. I had probably hidden 100 behind a function call, inhibiting the front end's checking. Or more likely behind something else. I have to look into something. > Why can't a backend simply generate code for the large shift, as if it had been a call to Word.Shift(..., 100)? It would be dead/unreachable code, but not a big deal. It looks like the front end skips the code when it can figure it out. I'll have to look into this more, but it is acting different/better than I realized. [Jay] EVAL -FIRST(INTEGER); [Jay] I believe the frontend should issue a warning. [Tony] Why? The front-end does not reason about overflow (except when computing compile-time constants). Overflow is a run-time concept!!!!!!!!!!!!! Because it can often easily prove that overflow will occur. It is worth warning about? if I write: a := 1 + 2; does the front end not optimize that to: a := 3; ? Imho it should 1 + 2 provably at compile time does not overflow. > how to enable overflow > The correct thing to do is introduce different interfaces/modules/types/functions > which either always do overflow checking, or, perhaps but less likely, > new interfaces/modules/types/functions that are runtime configurable, as > INTEGER was originally speced. > NOOOOO!!!!!!! That will impose an undue expense on targets where such checking is expensive. I doubt I suggested what you think I did. If someone really needs overflow checking, then it will cost them whatever it costs them, on whatever target they care about. I am NOT suggesting changing any existing interface/module/type. In fact I am proposing that FloatMode's hypothetical feature to enable overflow checking be removed. That "a + b", "a * b" etc., where a is INTEGER, never ever get overflow checking. If anyone needs it, they'd use some as yet to be specified and implemented type/interface/module. Like INTERFACE IntOv; TYPE T = INTEGER; PROCEDURE Add(a, b: T) RAISES Something: T; etc. or maybe a new compiler-builtin type INTEGEROV. Maybe use the word "safe" as it is popular for this sort of thing (search the web for "safeint"). - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From Highjinks at gmx.com Thu Mar 4 22:29:36 2010 From: Highjinks at gmx.com (Chris) Date: Thu, 4 Mar 2010 22:29:36 +0100 (CET) Subject: [M3devel] Referencing opaque C structs in M3 code? Message-ID: <20100304173345.06cb5d69.Highjinks@gmx.com> Alright, this has thrown me for a bit of a loop...no pun intended... Suppose I'm linking my Modula 3 code with a C library where the public API has a declaration like this... typedef struct opaque_foo_t opaque_foo_t; And the type opaque_foo_t is defined in the private part of the library. Do I need to create a seperate representation of the structure for my Modula3 program or can I just create a pointer for it and pass the pointer around? Assuming of course I dont have to actually reference anything inside opaque_foo_t. Would I just create an UNTRACED REF Ctypes.void_star or something similiar? I'm a little puzzled. Any help would be appreciated. -- Chris From mika at async.async.caltech.edu Thu Mar 4 22:49:17 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Thu, 04 Mar 2010 13:49:17 -0800 Subject: [M3devel] Referencing opaque C structs in M3 code? In-Reply-To: <20100304173345.06cb5d69.Highjinks@gmx.com> References: <20100304173345.06cb5d69.Highjinks@gmx.com> Message-ID: <20100304214917.9D0DD1A2080@async.async.caltech.edu> Ctypes.void_star is already what you want. It's supposed to behave like a (void *), which I think is what you want? You can also use ADDRESS but it's a bit less clear, I think. Mika Chris writes: >Alright, this has thrown me for a bit of a loop...no pun intended... > >Suppose I'm linking my Modula 3 code with a C library where the public API has > a declaration like this... > >typedef struct opaque_foo_t opaque_foo_t; > >And the type opaque_foo_t is defined in the private part of the library. > >Do I need to create a seperate representation of the structure for my Modula3 >program or can I just create a pointer for it and pass the pointer around? Ass >uming of course I dont have to actually reference anything inside opaque_foo_t >. Would I just create an UNTRACED REF Ctypes.void_star or something similiar? > >I'm a little puzzled. Any help would be appreciated. > > > >-- >Chris From hosking at cs.purdue.edu Thu Mar 4 22:59:42 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 4 Mar 2010 16:59:42 -0500 Subject: [M3devel] Referencing opaque C structs in M3 code? In-Reply-To: <20100304173345.06cb5d69.Highjinks@gmx.com> References: <20100304173345.06cb5d69.Highjinks@gmx.com> Message-ID: <7CAE16E7-C170-4A0B-B515-2B2DD5493DBE@cs.purdue.edu> You can assume that ADDRESS is entirely interchangeable with a C void *. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 4 Mar 2010, at 16:29, Chris wrote: > Alright, this has thrown me for a bit of a loop...no pun intended... > > Suppose I'm linking my Modula 3 code with a C library where the public API has a declaration like this... > > typedef struct opaque_foo_t opaque_foo_t; > > And the type opaque_foo_t is defined in the private part of the library. > > Do I need to create a seperate representation of the structure for my Modula3 program or can I just create a pointer for it and pass the pointer around? Assuming of course I dont have to actually reference anything inside opaque_foo_t. Would I just create an UNTRACED REF Ctypes.void_star or something similiar? > > I'm a little puzzled. Any help would be appreciated. > > > > -- > Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 5 02:00:37 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 5 Mar 2010 01:00:37 +0000 Subject: [M3devel] optimizing for size or speed? In-Reply-To: References: , , <20100211120309.GB27402@topoi.pooq.com>, Message-ID: I still find these decisions difficult. Esp. when the option is to call a function or not. Esp. for "builtin" stuff like 64bit math and set operations. - Jay ________________________________ > From: jay.krell at cornell.edu > To: hendrik at topoi.pooq.com; m3devel at elegosoft.com > Date: Thu, 11 Feb 2010 12:41:12 +0000 > Subject: Re: [M3devel] optimizing for size or speed? > > > > > > > > > cache -- good point I had forgotten, thanks. > > > > > > Still: > > use the divide instruction, which is smallest, or multiply by reciprocal (which is generally a multiply and a shift) > > (Given a 32x32=>64 multiply operation. x86 doesn't even have 32x32=>32, only 32x32=>64, I believe.) > > Any 32bit division by a constant can be optimized this way and every C compiler knows it. > > > > > > multiply by a constant using multiply instruction, or decompose into some adds? > > The AMD64 optimization guide suggests speed optimizations where they give a sequence for multiplication for every constant up to 32, some are just to use mul. But many are one or two other instructions. > > > > Multiply by 5 is: > > lea eax,[eax+eax*4] > > > > Multiply by 10 is: > > lea eax,[eax+eax*4] > > add eax,eax > > > > > The AMD64 manual even advises to inline 64bit shifts by a non-constant. > > But I can't get Visual C++ to do that. It always calls a function. > > > > > - Jay > > >> Date: Thu, 11 Feb 2010 07:03:09 -0500 >> From: hendrik at topoi.pooq.com >> To: m3devel at elegosoft.com >> Subject: Re: [M3devel] optimizing for size or speed? >> >> On Thu, Feb 11, 2010 at 08:53:08AM +0000, Jay K wrote: >>> >>> There are all kinds of equivalent code sequences. >>> For the maintainer of m3back to chose among. >> >> In case of doubt, go for size; size all by itself costs time in cache >> misses, paging, etc. >> >> Besides, it's possible to measure space. >> >> -- hendrik From jay.krell at cornell.edu Fri Mar 5 14:59:02 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 5 Mar 2010 13:59:02 +0000 Subject: [M3devel] examples In-Reply-To: <20100227235250.mbm07ehhwsokskg4@mail.elegosoft.com> References: , <20100227235250.mbm07ehhwsokskg4@mail.elegosoft.com> Message-ID: Olaf, it looks like I had it backwards. The release branch is as one would want it. You made the change there. Head lags. Should be ported to head but not pressing. Did the files have to move? Oh well. - Jay > Date: Sat, 27 Feb 2010 23:52:50 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] examples > > Quoting Jay K : > > > > > I think the release branch is missing the change to move examples. > > > > Update it? > Yes, please do. > > Olaf > > (I say this based on differences in the scripts.) > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 5 16:06:00 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 5 Mar 2010 15:06:00 +0000 Subject: [M3devel] release status? Message-ID: release status? - cvsupd hang is uninvestigated - NT386 lacks 64bit longint In head, 64bit longint support is quite good, however I found a bug recently, in head, that needs investigation; This code: C:\dev2\cm3.2\m3-sys\m3tests\src\p2\p227\Main.m3(129): result64 := Long.Insert(flipA * a64, flipB * b64, m, n); yields: C:\dev2\cm3.2\m3-sys\m3back\src\M3x86.m3(1045): u.Err("Couldn't find var to free in 'free_temp'"); - Also, head vs. release: TInt/TWord/m3front - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 5 16:36:57 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 5 Mar 2010 15:36:57 +0000 Subject: [M3devel] double double double double checking jmp_buf size/alignment In-Reply-To: <20100106162558.7B4101A2079@async.async.caltech.edu> References: , <20100106162558.7B4101A2079@async.async.caltech.edu> Message-ID: Mika, Hm. I would have expected 44 for FreeBSD/x86. Needs slightly more investigation. PPC_DARWIN we agree on. Thanks, - Jay > To: jay.krell at cornell.edu > Date: Wed, 6 Jan 2010 08:25:58 -0800 > From: mika at async.async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] double double double double checking jmp_buf size/alignment > > Forgot the #endif there? > > (81)trs80:~>./a.out > 48 4 > (82)trs80:~>uname -a > FreeBSD trs80.async.caltech.edu 4.11-RELEASE FreeBSD 4.11-RELEASE #3: Mon Nov 21 20:27:08 PST 2005 root at trs80.async.caltech.edu:/usr/src/sys/compile/TRS80 i386 > > [lapdog:~] mika% cc jay.c > [lapdog:~] mika% ./a.out > 768 4 > [lapdog:~] mika% uname -a > Darwin lapdog 8.11.0 Darwin Kernel Version 8.11.0: Wed Oct 10 18:26:00 PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC Power Macintosh powerpc > [lapdog:~] mika% > > Jay K writes: > >--_b0fc625a-8a60-4f2b-9394-0f52cc975894_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >Was truncated!..edited slightly to avoid.. > > > > > >From: jay.krell at cornell.edu > >To: m3devel at elegosoft.com > >Subject: double double double double checking jmp_buf size/alignment > >Date: Wed=2C 6 Jan 2010 12:28:02 +0000 > > > > > > > > > > > > > > > > > >Getting this right is very important. > > Well=2C we can overstate but it is wasteful. > > > > > >So anyone with any time=2C please compile/run this and send the "platform" = > >(uname -a) and the output=2C thanks. > >Or look at m3-libs/m3core/src/C/*/Csetjmp.i3 and > >m3-sys/m3middle/src/Target.m3 and see if all three agree. > > > > > >I have checked a bunch of systems myself but extra checking is good. > > > > > >If you have a system we do not yet or any longer support=2C those are ok to= > >o. > > (e.g. Alpha_*=2C ARM_*=2C MIPS_*=2C *_Irix=2C *_VMS=2C *_Tru64 etc.) > > > >=20 > >#include > >#include > > > >#ifdef __GNUC__ > >#define ALIGN_OF(x) ((int)(sizeof(struct { char a=3B x b=3B }) - sizeof(x))= > >) > >#else > >#define ALIGN_OF(x) ((int)__alignof(x)) > > > >int main() > >{ > > printf("%d %d\n"=2C (int)sizeof(jmp_buf)=2C ALIGN_OF(jmp_buf))=3B > > return 0=3B > >} > > > > > >Thanks=2C > > - Jay > > = > > > >--_b0fc625a-8a60-4f2b-9394-0f52cc975894_ > >Content-Type: text/html; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > > > > > > > > > >Was truncated!..edited slightly to avoid..


>g">From: jay.krell at cornell.edu
To: m3devel at elegosoft.com
Subject: dou= > >ble double double double checking jmp_buf size/alignment
Date: Wed=2C 6 = > >Jan 2010 12:28:02 +0000

> > > > > > > > > > > > > >Getting this right is very important.
 =3B Well=2C we can overstate = > >but it is wasteful.


So anyone with any time=2C please compile/ru= > >n this and send the "platform" (uname -a) and the output=2C thanks.
Or l= > >ook at m3-libs/m3core/src/C/*/Csetjmp.i3 and
m3-sys/m3middle/src/Target.= > >m3 and see if all three agree.


I have checked a bunch of systems= > > myself but extra checking is good.


If you have a system we do n= > >ot yet or any longer support=2C those are ok too.
 =3B(e.g. Alpha_*= > >=2C ARM_*=2C MIPS_*=2C *_Irix=2C *_VMS=2C *_Tru64 etc.)

 =3B
= > >#include <=3Bsetjmp.h>=3B
#include <=3Bstdio.h>=3B

#ifdef= > > __GNUC__
#define ALIGN_OF(x) ((int)(sizeof(struct { char a=3B x b=3B })= > > - sizeof(x)))
#else
#define ALIGN_OF(x) ((int)__alignof(x))

i= > >nt main()
{
 =3B printf("%d %d\n"=2C (int)sizeof(jmp_buf)=2C ALIG= > >N_OF(jmp_buf))=3B
 =3B return 0=3B
}


Thanks=2C
&nbs= > >p=3B- Jay
> >= > > > >--_b0fc625a-8a60-4f2b-9394-0f52cc975894_-- -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 5 16:40:28 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 5 Mar 2010 15:40:28 +0000 Subject: [M3devel] double double double double checking jmp_buf size/alignment In-Reply-To: References: , Message-ID: Randy, thanks, we agree. Target.m3: | Systems.I386_INTERIX => (* Visual C++'s 16, plus two ints, one to say if sigmask saved, and one to possibly save it. *) Jumpbuf_size := 18 * Address.size; | Systems.NT386, Systems.NT386GNU, Systems.I386_NT, Systems.I386_CYGWIN, Systems.I386_MINGW => (* Cygwin is 13, Visual C++ is 16. Interix is 18. Use 18 for interop. Note that Cygwin's setjmp.h header is wrong, off by a factor of 4. Cygwin provides setjmp and _setjmp that resolve the same. Visual C++ provides only _setjmp. Visual C++ also has _setjmp3 that the compiler generates a call to. In fact _setjmp appears to only use 8 ints and _setjmp3 appears to use more. Consider switching to _setjmp3. *) Jumpbuf_size := (18 * Address.size); - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 6 Jan 2010 15:03:02 -0500 Subject: Re: [M3devel] double double double double checking jmp_buf size/alignment Jay: Results on the following platforms are all identical: Windows 2000, Intel Pentium 3: 64 4 Windows XP, Intel Core Duo T2400: 64 4 Windows Vista, Intel Core2 Duo P9600: 64 4 Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, January 06, 2010 8:18 AM To: m3devel Subject: Re: [M3devel] double double double double checking jmp_buf size/alignment Was truncated!..edited slightly to avoid.. From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: double double double double checking jmp_buf size/alignment Date: Wed, 6 Jan 2010 12:28:02 +0000 Getting this right is very important. Well, we can overstate but it is wasteful. So anyone with any time, please compile/run this and send the "platform" (uname -a) and the output, thanks. Or look at m3-libs/m3core/src/C/*/Csetjmp.i3 and m3-sys/m3middle/src/Target.m3 and see if all three agree. I have checked a bunch of systems myself but extra checking is good. If you have a system we do not yet or any longer support, those are ok too. (e.g. Alpha_*, ARM_*, MIPS_*, *_Irix, *_VMS, *_Tru64 etc.) #include #include #ifdef __GNUC__ #define ALIGN_OF(x) ((int)(sizeof(struct { char a; x b; }) - sizeof(x))) #else #define ALIGN_OF(x) ((int)__alignof(x)) int main() { printf("%d %d\n", (int)sizeof(jmp_buf), ALIGN_OF(jmp_buf)); return 0; } Thanks, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Fri Mar 5 21:38:28 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Fri, 5 Mar 2010 15:38:28 -0500 Subject: [M3devel] optimizing for size or speed? In-Reply-To: References: Message-ID: <20100305203828.GA17524@topoi.pooq.com> On Fri, Mar 05, 2010 at 01:00:37AM +0000, Jay K wrote: > > I still find these decisions difficult. > Esp. when the option is to call a function or not. > Esp. for "builtin" stuff like 64bit math and set operations. > > - Jay That's right. They can be difficult. -- hendrik From hendrik at topoi.pooq.com Fri Mar 5 22:03:10 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Fri, 5 Mar 2010 16:03:10 -0500 Subject: [M3devel] Referencing opaque C structs in M3 code? In-Reply-To: <7CAE16E7-C170-4A0B-B515-2B2DD5493DBE@cs.purdue.edu> References: <20100304173345.06cb5d69.Highjinks@gmx.com> <7CAE16E7-C170-4A0B-B515-2B2DD5493DBE@cs.purdue.edu> Message-ID: <20100305210310.GB17524@topoi.pooq.com> On Thu, Mar 04, 2010 at 04:59:42PM -0500, Tony Hosking wrote: > You can assume that ADDRESS is entirely interchangeable with a C void *. What struct opaque_foo_t provides that void doesn't is some further type-checking -- struct opaque_foo_t is a different type from struct other_foo_t. Presumably you should use Modula 3 techniques for making an opaque types in your interface. - hendrik > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 4 Mar 2010, at 16:29, Chris wrote: > > > Alright, this has thrown me for a bit of a loop...no pun intended... > > > > Suppose I'm linking my Modula 3 code with a C library where the public API has a declaration like this... > > > > typedef struct opaque_foo_t opaque_foo_t; > > > > And the type opaque_foo_t is defined in the private part of the library. > > > > Do I need to create a seperate representation of the structure for my Modula3 program or can I just create a pointer for it and pass the pointer around? Assuming of course I dont have to actually reference anything inside opaque_foo_t. Would I just create an UNTRACED REF Ctypes.void_star or something similiar? > > > > I'm a little puzzled. Any help would be appreciated. > > > > > > > > -- > > Chris > From hendrik at topoi.pooq.com Sat Mar 6 07:22:08 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Sat, 6 Mar 2010 01:22:08 -0500 Subject: [M3devel] RC4 Message-ID: <20100306062208.GA16974@topoi.pooq.com> I installed a few packages of RC4 a while ago for Debiaan Squeeze Linux on intel 32-bit, and have had no problems with them. They work just fine, I installed the core package, and then gui and m3gdb. WHile installing I noticed the following: The installation instructions told me to use a new empty directory to unpack packages. Is it a new empty directory for each package, or can I use the same one for all packages? If I use the same one, should I empty it before each package? And should this be different from the directory where I unpacked cm3-bin-ws-m3devtool-TARGET-VERSION.tgz? I made a new directory to unpack each package, but the instructions should be more explicit. There should be some mention which packages depend on which others. I suspect, for example, that juno requires gui. Such constraints should be mentioned in the installatioin instructions. -- hendrik From hendrik at topoi.pooq.com Sat Mar 6 07:28:06 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Sat, 6 Mar 2010 01:28:06 -0500 Subject: [M3devel] FW: optimizing range checks? In-Reply-To: References: <20100302125931.619312474001@birch.elegosoft.com> Message-ID: <20100306062806.GB16974@topoi.pooq.com> On Tue, Mar 02, 2010 at 01:16:03PM +0000, Jay K wrote: > > Hey that's pretty clever. > > It costs a register, but given: > > > > if (b >= constantX|| b <= -constantY) > a = 0; > > > > The C compiler instead does something like: > > if ((b + constantY - 1) > (constantX + constantY - 1)) > > a = 0; > > > > This is something the front end could do in many cases.? > > Adding a constant to eliminate one of the compares and branches is a win. > > If an x86 compiler will give up a register for this, then it is probably a win everywhere. > > > > Granted, it probably requires silent overflow. Oh well. It requires unsigned arithmetic; in particular the final compare has to be unsigned. -- hendrik From hosking at cs.purdue.edu Sat Mar 6 16:12:47 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sat, 6 Mar 2010 10:12:47 -0500 Subject: [M3devel] Referencing opaque C structs in M3 code? In-Reply-To: <20100305210310.GB17524@topoi.pooq.com> References: <20100304173345.06cb5d69.Highjinks@gmx.com> <7CAE16E7-C170-4A0B-B515-2B2DD5493DBE@cs.purdue.edu> <20100305210310.GB17524@topoi.pooq.com> Message-ID: <20C86CDF-49BA-4CA8-918A-D048F110BB7F@cs.purdue.edu> You can have a TYPE foo = BRANDED "foo" ADDRESS to distinguish from other void types. On 5 Mar 2010, at 16:03, hendrik at topoi.pooq.com wrote: > On Thu, Mar 04, 2010 at 04:59:42PM -0500, Tony Hosking wrote: >> You can assume that ADDRESS is entirely interchangeable with a C void *. > > What struct opaque_foo_t provides that void doesn't is some further > type-checking -- struct opaque_foo_t is a different type from struct > other_foo_t. > > Presumably you should use Modula 3 techniques for making an opaque > types in your interface. > > - hendrik > >> >> Antony Hosking | Associate Professor | Computer Science | Purdue University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> On 4 Mar 2010, at 16:29, Chris wrote: >> >>> Alright, this has thrown me for a bit of a loop...no pun intended... >>> >>> Suppose I'm linking my Modula 3 code with a C library where the public API has a declaration like this... >>> >>> typedef struct opaque_foo_t opaque_foo_t; >>> >>> And the type opaque_foo_t is defined in the private part of the library. >>> >>> Do I need to create a seperate representation of the structure for my Modula3 program or can I just create a pointer for it and pass the pointer around? Assuming of course I dont have to actually reference anything inside opaque_foo_t. Would I just create an UNTRACED REF Ctypes.void_star or something similiar? >>> >>> I'm a little puzzled. Any help would be appreciated. >>> >>> >>> >>> -- >>> Chris >> From jay.krell at cornell.edu Sun Mar 7 10:12:38 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 7 Mar 2010 09:12:38 +0000 Subject: [M3devel] word.insert/extract warnings/optimizations Message-ID: IO.PutInt(Word.Insert(a,b,20,20)); IO.PutInt(Word.Extract(1,1,50)); IO.PutInt(Word.Extract(1,30,30)); Tony, there's discrepancy between Word.Extract and Word.Insert. m3front does a better job of checking and optimizing Word.Extract. For Insert it always does a runtime add, and a runtime range check against the number of bits in integer. It isn't wrong, just could be better. For Extract, it checks either constant against the number of bits, if they are both constant, it checks the sum and calls extract_mn. If they are both constant and the sum is too large, it just generates code to cause a range fault at runtime check_hi(1 vs. 0). Probably that could be a little more direct. If one is constant it calls extract_n, or a slightly optimized use of extract if the other is constant. I'll probably tackle this soon. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 7 12:25:10 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 7 Mar 2010 11:25:10 +0000 Subject: [M3devel] -no-m3ship-resolution Message-ID: This is head. Two problems here. One the feature apparently not fully working. I should look at that. Two the test is too platform specific -- lib.lib vs. liblib.a. Maybe reasonable to support some text substitution over the results to address that. Or heck, maybe abstract out the "naming conventions"? Maybe too difficult and too little gain. I'm surprised even have TARGET like that. The main point I think is the install root. --- ../src/p2/p221/stdout.build 2009-12-15 03:04:01.000000000 -0800 +++ ../src/p2/p221/NT386/stdout.build 2010-03-07 03:11:39.531250000 -0800 @@ -1,7 +1,7 @@ -make_dir(PKG_INSTALL & "/p221/" & TARGET) -install_file(".M3EXPORTS", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file("liblib.a", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file("liblib.m3x", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file(".M3WEB", PKG_INSTALL & "/p221/" & TARGET, "0644") -make_dir(PKG_INSTALL & "/p221") -install_file("../Main.m3", PKG_INSTALL & "/p221", "0644") +make_dir("/cm3/pkg/p221/" & TARGET) +install_file(".M3EXPORTS", "/cm3/pkg/p221/" & TARGET, "0644") +install_file("lib.lib", "/cm3/pkg/p221/" & TARGET, "0644") +install_file("lib.m3x", "/cm3/pkg/p221/" & TARGET, "0644") +install_file(".M3WEB", "/cm3/pkg/p221/" & TARGET, "0644") +make_dir("/cm3/pkg/p221") +install_file("../Main.m3", "/cm3/pkg/p221", "0644") --- p222 --- .M3SHIP Library - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 7 12:48:59 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 7 Mar 2010 11:48:59 +0000 Subject: [M3devel] -no-m3ship-resolution In-Reply-To: References: Message-ID: My CM3_INSTALL environment variable has forward slashes. It'd probably work otherwise. One fix is: Index: M3Build.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/cm3/src/M3Build.m3,v retrieving revision 1.31 diff -u -r1.31 M3Build.m3 --- M3Build.m3 4 Sep 2009 10:25:26 -0000 1.31 +++ M3Build.m3 7 Mar 2010 11:40:50 -0000 @@ -133,7 +133,7 @@ (* some more config dependent backward compatibility hacks... *) Quake.Define (t, "M3SEARCH_TABLES", "-T" & M3TFile); Quake.Define (t, "DEFAULT_BUILD_DIR", GetConfig (t, "BUILD_DIR")); - Quake.Define (t, "M3", M3Path.New (GetConfig (t, "BIN_USE"), "cm3")); + Quake.Define (t, "M3", TextUtils.Substitute(M3Path.New (GetConfig (t, "BIN_USE"), "cm3"), "/", M3Path.SlashText)); Quake.Define (t, "PACKAGE_DIR", pkg); t.build_pkg := M3ID.Add (Pathname.Last (pkg)); @@ -143,19 +143,19 @@ (* M3Path.New is used to canonicalize the paths -- to remove dots *) - t.pkg_use := M3Path.New (GetConfig (t, "PKG_USE")); + t.pkg_use := TextUtils.Substitute(M3Path.New (GetConfig (t, "PKG_USE")), "/", M3Path.SlashText); (* not in Quake.Machine - t.bin_use := M3Path.New (GetConfig (t, "BIN_USE")); - t.lib_use := M3Path.New (GetConfig (t, "LIB_USE")); + t.bin_use := TextUtils.Substitute(M3Path.New (GetConfig (t, "BIN_USE")), "/", M3Path.SlashText); + t.lib_use := TextUtils.Substitute(M3Path.New (GetConfig (t, "LIB_USE")), "/", M3Path.SlashText); *) - t.pkg_install := M3Path.New (GetConfig (t, "PKG_INSTALL")); - t.install_root := M3Path.New (GetConfig (t, "INSTALL_ROOT")); - t.bin_install := M3Path.New (GetConfig (t, "BIN_INSTALL")); - t.lib_install := M3Path.New (GetConfig (t, "LIB_INSTALL")); - t.emacs_install := M3Path.New (GetConfig (t, "EMACS_INSTALL")); - t.doc_install := M3Path.New (GetConfig (t, "DOC_INSTALL")); - t.man_install := M3Path.New (GetConfig (t, "MAN_INSTALL")); - t.html_install := M3Path.New (GetConfig (t, "HTML_INSTALL")); + t.pkg_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "PKG_INSTALL")), "/", M3Path.SlashText); + t.install_root := TextUtils.Substitute(M3Path.New (GetConfig (t, "INSTALL_ROOT")), "/", M3Path.SlashText); + t.bin_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "BIN_INSTALL")), "/", M3Path.SlashText); + t.lib_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "LIB_INSTALL")), "/", M3Path.SlashText); + t.emacs_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "EMACS_INSTALL")), "/", M3Path.SlashText); + t.doc_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "DOC_INSTALL")), "/", M3Path.SlashText); + t.man_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "MAN_INSTALL")), "/", M3Path.SlashText); + t.html_install := TextUtils.Substitute(M3Path.New (GetConfig (t, "HTML_INSTALL")), "/", M3Path.SlashText); t.have_pkgtools := GetConfigBool (t, "HAVE_PKGTOOLS"); t.at_SRC := GetConfigBool (t, "AT_SRC"); t.system_liborder := QVal.ToArray (t, ConfigDefn (t, "SYSTEM_LIBORDER").value); but I'm not sure I like that. I think probably we should have parallel variables bin_install_forwardslash, etc., in which \\ is replaced by /. If we compute the paths ourselves on Win32, we use \\. But if user sets them, we leave them alone. Either slash works. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Sun, 7 Mar 2010 11:25:10 +0000 Subject: [M3devel] -no-m3ship-resolution This is head. Two problems here. One the feature apparently not fully working. I should look at that. Two the test is too platform specific -- lib.lib vs. liblib.a. Maybe reasonable to support some text substitution over the results to address that. Or heck, maybe abstract out the "naming conventions"? Maybe too difficult and too little gain. I'm surprised even have TARGET like that. The main point I think is the install root. --- ../src/p2/p221/stdout.build 2009-12-15 03:04:01.000000000 -0800 +++ ../src/p2/p221/NT386/stdout.build 2010-03-07 03:11:39.531250000 -0800 @@ -1,7 +1,7 @@ -make_dir(PKG_INSTALL & "/p221/" & TARGET) -install_file(".M3EXPORTS", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file("liblib.a", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file("liblib.m3x", PKG_INSTALL & "/p221/" & TARGET, "0644") -install_file(".M3WEB", PKG_INSTALL & "/p221/" & TARGET, "0644") -make_dir(PKG_INSTALL & "/p221") -install_file("../Main.m3", PKG_INSTALL & "/p221", "0644") +make_dir("/cm3/pkg/p221/" & TARGET) +install_file(".M3EXPORTS", "/cm3/pkg/p221/" & TARGET, "0644") +install_file("lib.lib", "/cm3/pkg/p221/" & TARGET, "0644") +install_file("lib.m3x", "/cm3/pkg/p221/" & TARGET, "0644") +install_file(".M3WEB", "/cm3/pkg/p221/" & TARGET, "0644") +make_dir("/cm3/pkg/p221") +install_file("../Main.m3", "/cm3/pkg/p221", "0644") --- p222 --- .M3SHIP Library - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 7 14:37:28 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 7 Mar 2010 13:37:28 +0000 Subject: [M3devel] test p078 works in release but not head -- NUMBER(open array constant) not constant Message-ID: C:\dev2\cm3.2\m3-sys\m3tests\src\p0\p078 This compiles with release but not head. (* Copyright (C) 1994, Digital Equipment Corporation. *) (* All rights reserved. *) (* See the file COPYRIGHT for a full description. *) MODULE Main; FROM Test IMPORT done, checkI, checkB; CONST Numbers = ARRAY OF INTEGER {2, 3, 5, 7, 11}; First = Numbers [FIRST (Numbers)]; Last = Numbers [LAST (Numbers)]; Number = NUMBER (Numbers); Empty = ARRAY OF INTEGER {}; EFirst = FIRST (Empty); ELast = LAST (Empty); ENumber = NUMBER (Empty); BEGIN checkI (First, 2); checkI (Last, 11); checkI (Number, 5); checkB (EFirst > ELast, TRUE); checkI (ENumber, 0); done (); END Main. new source -> compiling Main.m3 "..\Main.m3", line 14: value is not constant "..\Main.m3", line 19: value is not constant 2 errors encountered compilation failed => not building program "pgm.exe" Fatal Error: package build failed C:\dev2\cm3.2\m3-sys\m3tests\src\p0\p078> It is the lines with "NUMBER". - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 7 15:14:29 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 7 Mar 2010 14:14:29 +0000 Subject: [M3devel] test p078 works in release but not head -- NUMBER(open array constant) not constant In-Reply-To: References: Message-ID: nevermind. Index: Number.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/builtinOps/Number.m3,v retrieving revision 1.5 diff -u -r1.5 Number.m3 --- Number.m3 18 Feb 2010 02:33:09 -0000 1.5 +++ Number.m3 7 Mar 2010 14:11:17 -0000 @@ -102,8 +102,8 @@ IF ArrayExpr.GetBounds (e, min, max) AND TInt.Subtract (max, min, tmp) AND TInt.Add (tmp, TInt.One, num) - AND NOT TInt.LT (num, Target.Integer.max) - AND NOT TInt.LT (Target.Integer.max, num) + AND TInt.GE (num, Target.Integer.min) + AND TInt.LE (num, Target.Integer.max) THEN RETURN IntegerExpr.New (Int.T, num); ELSE RETURN NIL; END; Notice how the full assortment of LE/GE/LT/GT/EQ/NE makes it easier to read and write the code and get it correct. :) Of course, the "double max" stands out as incorrect either way. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu; m3devel at elegosoft.com Date: Sun, 7 Mar 2010 13:37:28 +0000 Subject: [M3devel] test p078 works in release but not head -- NUMBER(open array constant) not constant C:\dev2\cm3.2\m3-sys\m3tests\src\p0\p078 This compiles with release but not head. (* Copyright (C) 1994, Digital Equipment Corporation. *) (* All rights reserved. *) (* See the file COPYRIGHT for a full description. *) MODULE Main; FROM Test IMPORT done, checkI, checkB; CONST Numbers = ARRAY OF INTEGER {2, 3, 5, 7, 11}; First = Numbers [FIRST (Numbers)]; Last = Numbers [LAST (Numbers)]; Number = NUMBER (Numbers); Empty = ARRAY OF INTEGER {}; EFirst = FIRST (Empty); ELast = LAST (Empty); ENumber = NUMBER (Empty); BEGIN checkI (First, 2); checkI (Last, 11); checkI (Number, 5); checkB (EFirst > ELast, TRUE); checkI (ENumber, 0); done (); END Main. new source -> compiling Main.m3 "..\Main.m3", line 14: value is not constant "..\Main.m3", line 19: value is not constant 2 errors encountered compilation failed => not building program "pgm.exe" Fatal Error: package build failed C:\dev2\cm3.2\m3-sys\m3tests\src\p0\p078> It is the lines with "NUMBER". - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Mon Mar 8 10:05:47 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 08 Mar 2010 10:05:47 +0100 Subject: [M3devel] RC4 In-Reply-To: <20100306062208.GA16974@topoi.pooq.com> References: <20100306062208.GA16974@topoi.pooq.com> Message-ID: <20100308100547.2ytic2gw8wcwwo0k@mail.elegosoft.com> Quoting hendrik at topoi.pooq.com: > I installed a few packages of RC4 a while ago for Debiaan Squeeze Linux > on intel 32-bit, and have had no problems with them. They work just > fine, > > I installed the core package, and then gui and m3gdb. Fine. > WHile installing I noticed the following: > > The installation instructions told me to use a new empty > directory to unpack packages. Is it a new empty directory for each > package, or can I use the same one for all packages? If I use the > same one, should I empty it before each package? And should this be > different from the directory where I unpacked > cm3-bin-ws-m3devtool-TARGET-VERSION.tgz? I made a new directory to > unpack each package, but the instructions should be more explicit. I'll try to improve that for RC5. > There should be some mention which packages depend on which others. I > suspect, for example, that juno requires gui. Such constraints should > be mentioned in the installatioin instructions. You mean something like http://www.modula3.com/cm3/releng/collection-juno.html ? Probably the link to the package descriptions is too hidden in the table (last column). Thanks for the feedback, Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Mon Mar 8 10:15:59 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 8 Mar 2010 09:15:59 +0000 Subject: [M3devel] m3cg.i3 reduction? Message-ID: The backend interface has a few aspects that the frontend does not use. Implementations of these are therefore extremely difficult to test and therefore probably don't work. I propose: extract and extract_n should not take a sign_extend:BOOLEAN parameter. It is always false. extract_mn shall retain its sign_extend:BOOLEAN parameter. Though really, it could go too. The front end could just issue divide and the backends could probably figure it out. I like the frontend doing the work though. (Really, if it going to optimize divide by a power of 2, it might be nice to compute reciprocals too...on my list for m3back..) The integer parameters to extract*/insert* should be CARDINAL instead of INTEGER. Or [0..63], with the "63" abstracted out somehow and comments that clarify it is really sometimes only 0..31. The front end already issues range checks for these parameters. The backend shouldn't worry about such wide ranges. Arguably remove insert_m/n and extract_m/n and just have insert/extract. The NT386 backend already notices when parameters on the stack are constants, and the gcc backend probably does too, at least when optimizing. The "specializations" do make it easier for a hypothetical backend simpler than the NT386 to optimize a bit. So I'm on the fence there. zero_n should be removed. It isn't used. Far more radically, I'm suspicious that the use of a stack is a good idea. It'd probably be just as easy or easier, and lead to better code, to have a *sort* of union that encapsulates constants and variables, and pass each "operation" (add/multiply/subtract/insert/extract/etc.) its parameters with the function all. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 8 15:59:41 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 8 Mar 2010 09:59:41 -0500 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: References: Message-ID: <28E061D6-8DB0-4C69-A23D-F460573E16E0@cs.purdue.edu> On 8 Mar 2010, at 04:15, Jay K wrote: > The backend interface has a few aspects that the frontend does not use. > Implementations of these are therefore extremely difficult to test and therefore probably don't work. This is inevitable. Your backend is not implementing an interface that the front-end defines. It is implementing an interface defined in m3middle which is much wider than that used by m3front. For good reason, we should not dumb down m3middle just because m3front doesn't make use of all its operations. Similarly, you should not assume that m3front will not make use of some operations in future. This is good practice for separation of concerns in support of modularity. > I propose: > > > extract and extract_n should not take a sign_extend:BOOLEAN parameter. > It is always false. Only as currently generated by m3front. > extract_mn shall retain its sign_extend:BOOLEAN parameter. For consistency and generality we should retain it for all extract operators. > Though really, it could go too. > The front end could just issue divide and the backends could probably > figure it out. I like the frontend doing the work though. The front-end should not be concerned with optimisation. It's job is semantics, and having extract/insert is important for some targets. > (Really, if it going to optimize divide by a power of 2, it might be nice > to compute reciprocals too...on my list for m3back..) Same. Front-ends should not be worried about optimising. > The integer parameters to extract*/insert* should be CARDINAL instead of INTEGER. > Or [0..63], with the "63" abstracted out somehow and comments that clarify it is > really sometimes only 0..31. That is not a general interface. Just because m3front filters the operands it provides doesn't mean that all upstream clients of m3middle will do so. > The front end already issues range checks for these parameters. > The backend shouldn't worry about such wide ranges. The backend should be prepared for them. > Arguably remove insert_m/n and extract_m/n and just have insert/extract. That is a possibility. But again, some backends might find it convenient to know about constant operands. Let's not constrain things. > The NT386 backend already notices when parameters on the stack are constants, > and the gcc backend probably does too, at least when optimizing. > The "specializations" do make it easier for a hypothetical backend simpler > than the NT386 to optimize a bit. Precisely! > So I'm on the fence there. We should retain them. > Zero_n should be removed. I've considered using it for some initialisation support. We don't want to throw something out that may later be useful. M3CG was designed for generality, and actually even predates the Modula-3 language itself. It is derived from code generators at DEC SRC from the 1980s. > It isn't used. Yet... > Far more radically, I'm suspicious that the use of a stack is a good idea. > It'd probably be just as easy or easier, and lead to better code, to > have a *sort* of union that encapsulates constants and variables, > and pass each "operation" (add/multiply/subtract/insert/extract/etc.) its > parameters with the function all. Stack models versus "registers" have always been argued about in compiling. Witness the Java VM spec's use of a stack as compared to the fact that most Java VMs convert the stack to registers for specific targets. > > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Mar 8 17:17:24 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 8 Mar 2010 16:17:24 +0000 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: <28E061D6-8DB0-4C69-A23D-F460573E16E0@cs.purdue.edu> References: , <28E061D6-8DB0-4C69-A23D-F460573E16E0@cs.purdue.edu> Message-ID: > Front-ends should not be worried about optimising. But it does so much already. It unrolls comparisons of "solid" types to a certain extent. It took me a while to track that down. It converts divides by powers of two into shift/extract. m3front (or m3middle?) could act as "shared code" for multiple backends. Computing the reciprocal is something that would be common to various backends. The computation itself is portable. The backend could chose to use it or not (maybe some processor has a fast divide instruction..or equally slow divide and mutiply...). How about TInt.Reciprocal or somesuch? > doesn't mean that all upstream clients of m3middle will do so What other clients of m3middle could possibly exist? And that really need more functionality than m3front needs? What does insert/extract with negative numbers mean? I'll see if the .i3 files describes it in comments. Please let's not just invent general generalities, that we don't use, and can't test. What m3front needs, we should do. What m3front doesn't need, we should not do. m3middle serves m3front. If/when m3front needs it, do it then. Generalities produce dead untestable buggy code and wasted time. There was definitely some in m3back, e.g. sign extended extract seemed wrong for count = 0. I forgot another suggestion: set_eq and set_ne are implemented the same by the two backends, by calling memcmp. All the other comparisons are the same too, via calling functions. Maybe the frontend should just know about this and call the functions? These are unusually high level operations in the backends, that neither one implements in an at all clever way. - Jay From: hosking at cs.purdue.edu Date: Mon, 8 Mar 2010 09:59:41 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] m3cg.i3 reduction? On 8 Mar 2010, at 04:15, Jay K wrote: The backend interface has a few aspects that the frontend does not use. Implementations of these are therefore extremely difficult to test and therefore probably don't work. This is inevitable. Your backend is not implementing an interface that the front-end defines. It is implementing an interface defined in m3middle which is much wider than that used by m3front. For good reason, we should not dumb down m3middle just because m3front doesn't make use of all its operations. Similarly, you should not assume that m3front will not make use of some operations in future. This is good practice for separation of concerns in support of modularity. I propose: extract and extract_n should not take a sign_extend:BOOLEAN parameter. It is always false. Only as currently generated by m3front. extract_mn shall retain its sign_extend:BOOLEAN parameter. For consistency and generality we should retain it for all extract operators. Though really, it could go too. The front end could just issue divide and the backends could probably figure it out. I like the frontend doing the work though. The front-end should not be concerned with optimisation. It's job is semantics, and having extract/insert is important for some targets. (Really, if it going to optimize divide by a power of 2, it might be nice to compute reciprocals too...on my list for m3back..) Same. Front-ends should not be worried about optimising. The integer parameters to extract*/insert* should be CARDINAL instead of INTEGER. Or [0..63], with the "63" abstracted out somehow and comments that clarify it is really sometimes only 0..31. That is not a general interface. Just because m3front filters the operands it provides doesn't mean that all upstream clients of m3middle will do so. The front end already issues range checks for these parameters. The backend shouldn't worry about such wide ranges. The backend should be prepared for them. Arguably remove insert_m/n and extract_m/n and just have insert/extract. That is a possibility. But again, some backends might find it convenient to know about constant operands. Let's not constrain things. The NT386 backend already notices when parameters on the stack are constants, and the gcc backend probably does too, at least when optimizing. The "specializations" do make it easier for a hypothetical backend simpler than the NT386 to optimize a bit. Precisely! So I'm on the fence there. We should retain them. Zero_n should be removed. I've considered using it for some initialisation support. We don't want to throw something out that may later be useful. M3CG was designed for generality, and actually even predates the Modula-3 language itself. It is derived from code generators at DEC SRC from the 1980s. It isn't used. Yet... Far more radically, I'm suspicious that the use of a stack is a good idea. It'd probably be just as easy or easier, and lead to better code, to have a *sort* of union that encapsulates constants and variables, and pass each "operation" (add/multiply/subtract/insert/extract/etc.) its parameters with the function all. Stack models versus "registers" have always been argued about in compiling. Witness the Java VM spec's use of a stack as compared to the fact that most Java VMs convert the stack to registers for specific targets. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Mon Mar 8 13:25:36 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Mon, 8 Mar 2010 07:25:36 -0500 Subject: [M3devel] RC4 In-Reply-To: <20100308100547.2ytic2gw8wcwwo0k@mail.elegosoft.com> References: <20100306062208.GA16974@topoi.pooq.com> <20100308100547.2ytic2gw8wcwwo0k@mail.elegosoft.com> Message-ID: <20100308122536.GA6668@topoi.pooq.com> On Mon, Mar 08, 2010 at 10:05:47AM +0100, Olaf Wagner wrote: > Quoting hendrik at topoi.pooq.com: > > >I installed a few packages of RC4 a while ago for Debiaan Squeeze Linux > >on intel 32-bit, and have had no problems with them. They work just > >fine, > > > >I installed the core package, and then gui and m3gdb. > > Fine. > > >WHile installing I noticed the following: > > > >The installation instructions told me to use a new empty > >directory to unpack packages. Is it a new empty directory for each > >package, or can I use the same one for all packages? If I use the > >same one, should I empty it before each package? And should this be > >different from the directory where I unpacked > >cm3-bin-ws-m3devtool-TARGET-VERSION.tgz? I made a new directory to > >unpack each package, but the instructions should be more explicit. > > I'll try to improve that for RC5. > > >There should be some mention which packages depend on which others. I > >suspect, for example, that juno requires gui. Such constraints should > >be mentioned in the installatioin instructions. > > You mean something like > > http://www.modula3.com/cm3/releng/collection-juno.html Yes. Something like that! I've never seen that page before. > > ? Probably the link to the package descriptions is too hidden in the table > (last column). What table? > > Thanks for the feedback, > > Olaf It looks like a website navigability problem. As installer, I see the archive descriptions in http://www.modula3.com/cm3/releng/ where the Juno line says, cm3-bin-ws-juno-*.tgz source/binary, application, optional A constraint-based graphical editor and there isn't any mention of a package description. -- hendrik > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > From hendrik at topoi.pooq.com Mon Mar 8 13:37:46 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Mon, 8 Mar 2010 07:37:46 -0500 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: <28E061D6-8DB0-4C69-A23D-F460573E16E0@cs.purdue.edu> References: <28E061D6-8DB0-4C69-A23D-F460573E16E0@cs.purdue.edu> Message-ID: <20100308123746.GB6668@topoi.pooq.com> On Mon, Mar 08, 2010 at 09:59:41AM -0500, Tony Hosking wrote: > > M3CG was designed for generality, and actually even predates the > Modula-3 language itself. It is derived from code generators at DEC > SRC from the 1980s. I've been thinking for a while that the Modula 3 back end and a fair bit of its run-time system are probably good enough to be used by other garbage-collected languages. Almost no one else does a good job of combining parallel processing with garbage collection, for example. -- hendrik From rodney_bates at lcwb.coop Mon Mar 8 18:42:16 2010 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 08 Mar 2010 11:42:16 -0600 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: References: Message-ID: <4B9536F8.7000104@lcwb.coop> It is never realistic to try to thoroughly test a later (than the first) phase of any compiler solely by pumping source code through the earlier phases. Every compiler I have ever worked on had tools to allow hand-coded input to later phases, either already, or else I wrote one, if I had significant responsibility for testing. Sometimes it can be done with a plain old text editor, if the intermediate stream is character-encoded. Similarly, you need to be able dump the intermediate forms of output from earlier phases in human-readable form. Don't we already have at least some of this? Jay K wrote: > The backend interface has a few aspects that the frontend does not use. > Implementations of these are therefore /extremely/ difficult to test and > therefore probably don't work. > > > From hosking at cs.purdue.edu Mon Mar 8 20:32:53 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 8 Mar 2010 14:32:53 -0500 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: <4B9536F8.7000104@lcwb.coop> References: <4B9536F8.7000104@lcwb.coop> Message-ID: On 8 Mar 2010, at 12:42, Rodney M. Bates wrote: > It is never realistic to try to thoroughly test a later (than the first) phase of > any compiler solely by pumping source code through the earlier phases. Every > compiler I have ever worked on had tools to allow hand-coded input to later > phases, either already, or else I wrote one, if I had significant responsibility > for testing. Sometimes it can be done with a plain old text editor, if the > intermediate stream is character-encoded. Indeed. > Similarly, you need to be able dump the intermediate forms of output from earlier > phases in human-readable form. Don't we already have at least some of this? Yes, m3cgcat and friends... > > Jay K wrote: >> The backend interface has a few aspects that the frontend does not use. >> Implementations of these are therefore /extremely/ difficult to test and therefore probably don't work. >> From jay.krell at cornell.edu Tue Mar 9 03:03:27 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 02:03:27 +0000 Subject: [M3devel] m3cg.i3 reduction? In-Reply-To: References: , <4B9536F8.7000104@lcwb.coop>, Message-ID: There is no such mode currently for m3back. I think. At least it doesn't operate that way normally. It keeps very little in memory, going fairly directly from function calls to file output. Not entirely, but largely. Not a bad idea though. But maybe I'm wrong here. Maybe the thing to write/read to the gcc backend can also drive m3back? And then I can start with m3front output but edit in variations? That could be handy. Still, these things are give and take. You theorize as to what you might need. Then you write the consumer. Then you discover you might have missed some things and might have put in some unnecessary things. Not all generalizations should be kept. Interfaces should be reviewed for areas that ended up not needed. And you also write the producer and find some things to be easy or difficult. "Build it and they will come", to some extent yes, to some extent no. sign extension was "so easy" that the original authors wrote it, including for non-constants, but I don't think it works for count=0 for example, which the current front end never exercises..not even with constants.. Given my current implementation strategy, which produces less efficient but clear and dependency-free code, it is perhaps easier to extend things back. (still waiting to hear if the larger size bothers anyone, though the tables are gone, that is a reduction; not sure clarity is so important at the assembly level, it is assembly after all) - Jay ---------------------------------------- > From: hosking at cs.purdue.edu > Date: Mon, 8 Mar 2010 14:32:53 -0500 > To: rodney_bates at lcwb.coop > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] m3cg.i3 reduction? > > On 8 Mar 2010, at 12:42, Rodney M. Bates wrote: > >> It is never realistic to try to thoroughly test a later (than the first) phase of >> any compiler solely by pumping source code through the earlier phases. Every >> compiler I have ever worked on had tools to allow hand-coded input to later >> phases, either already, or else I wrote one, if I had significant responsibility >> for testing. Sometimes it can be done with a plain old text editor, if the >> intermediate stream is character-encoded. > > Indeed. > >> Similarly, you need to be able dump the intermediate forms of output from earlier >> phases in human-readable form. Don't we already have at least some of this? > > Yes, m3cgcat and friends... > >> >> Jay K wrote: >>> The backend interface has a few aspects that the frontend does not use. >>> Implementations of these are therefore /extremely/ difficult to test and therefore probably don't work. >>> > From jay.krell at cornell.edu Tue Mar 9 06:09:19 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 05:09:19 +0000 Subject: [M3devel] m3_build vs. build in parse.c? In-Reply-To: <16ADF5CA-B729-40CC-ADAB-5FA170C9084B@cs.purdue.edu> References: , <16ADF5CA-B729-40CC-ADAB-5FA170C9084B@cs.purdue.edu> Message-ID: I understand why use m3_build instead of build. But why ever use build? Just when m3_build has no chance of optimizing? Or an oversight? - Jay ________________________________ > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > CC: m3commit at elegosoft.com > Subject: RE: [M3commit] m3_build vs. build in parse.c? > Date: Thu, 4 Mar 2010 08:46:06 +0000 > > > Why not always call m3_build? > > Why ever call build? > > > > - Jay > >> From: hosking at cs.purdue.edu >> Date: Thu, 4 Mar 2010 02:58:30 -0500 >> To: jay.krell at cornell.edu >> CC: m3commit at elegosoft.com >> Subject: Re: [M3commit] m3_build vs. build in parse.c? >> >> So that we can avoid having to analyse all the constants ourselves. We can simply generate the trees and then have gcc fold them. >> >> On 4 Mar 2010, at 02:16, Jay K wrote: >> >>> Why use one vs. the other? >>> It appears that they are equivalent *except* that m3_build attempts to optimize, >>> but falls back to build if it can't. >>> >>> That is, m3_build calls fold_build. >>> >>> - Jay >>> >>> >>> >>> >> From hosking at cs.purdue.edu Tue Mar 9 06:33:07 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 9 Mar 2010 00:33:07 -0500 Subject: [M3devel] m3_build vs. build in parse.c? In-Reply-To: References: , <16ADF5CA-B729-40CC-ADAB-5FA170C9084B@cs.purdue.edu> Message-ID: <556CE042-9DBD-4432-BBD8-82E0A2DD5801@cs.purdue.edu> Not sure if it break anything or not, but no good reason I know of. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 9 Mar 2010, at 00:09, Jay K wrote: > > I understand why use m3_build instead of build. > But why ever use build? Just when m3_build has > no chance of optimizing? Or an oversight? > > - Jay > > ________________________________ >> From: jay.krell at cornell.edu >> To: hosking at cs.purdue.edu >> CC: m3commit at elegosoft.com >> Subject: RE: [M3commit] m3_build vs. build in parse.c? >> Date: Thu, 4 Mar 2010 08:46:06 +0000 >> >> >> Why not always call m3_build? >> >> Why ever call build? >> >> >> >> - Jay >> >>> From: hosking at cs.purdue.edu >>> Date: Thu, 4 Mar 2010 02:58:30 -0500 >>> To: jay.krell at cornell.edu >>> CC: m3commit at elegosoft.com >>> Subject: Re: [M3commit] m3_build vs. build in parse.c? >>> >>> So that we can avoid having to analyse all the constants ourselves. We can simply generate the trees and then have gcc fold them. >>> >>> On 4 Mar 2010, at 02:16, Jay K wrote: >>> >>>> Why use one vs. the other? >>>> It appears that they are equivalent *except* that m3_build attempts to optimize, >>>> but falls back to build if it can't. >>>> >>>> That is, m3_build calls fold_build. >>>> >>>> - Jay >>>> >>>> >>>> >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 9 15:25:48 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 09 Mar 2010 15:25:48 +0100 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT Message-ID: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com> some windows problems again, probably only help needed... any takers? Olaf ----- Forwarded message from bugs at elego.de ----- Date: Tue, 09 Mar 2010 02:55:44 -0000 From: CM3 Reply-To: CM3 Subject: [CM3] #1082: Windows NT To: @MISSING_DOMAIN #1082: Windows NT -------------------------------------+-------------------------------------- Reporter: ilikesci@? | Owner: wagner Type: sw-test | Status: new Priority: medium | Milestone: Component: misc | Version: 5.8-RC3 Severity: non-critical | Keywords: Relnote: | Org: Estimatedhours: 0 | Hours: 0 Billable: 0 | Totalhours: 0 Internal: 0 | -------------------------------------+-------------------------------------- Htr: Remove msys out of your path. Fix: Env: Microsoft Windows Vista -------------------------------------+-------------------------------------- I am new to modula-3 but have tried a few times to get cm3 working on my MS-Windows machines over the years. This is my new attempt at that. I downloaded the NT386 files and have them unpacked. I have put cm3 in e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is that when I ran cminstall it asked for a tar.exe, gzip.exe, and msys-1.0.dll. I just happened to have msys installed on my computer and dropped them in the folder and cminstall seemed to work. I just wanted to let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and so I ran it per the suggestion on your website. It asked a few questions about what browser I wanted to use and such. The browser does pop up on localhost:3800 but times out. The console spits out: Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 calling start_browser(http://localhost:3800/) starting TCP service start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" http://localhost:3800/ CM3-IDE is shutting down because start_browser() returned TRUE. TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept TCPServer: aborting... I am thinking maybe something else might be running on the port and causing a problem loading page. I copied the startReactor.cmd to the bin directory and tried it and got the following: ------------------------------------------------------------------------------- startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by RCC ------------------------------------------------------------------------------- FATAL ERROR: Unable to find CM3 installation. CM3_ROOT expected in folder C:\cm3 CM3_BIN expected in folder C:\cm3\bin CM3.EXE expected in file C:\cm3\bin\cm3.exe Reactor.EXE expected in file C:\cm3\bin\reactor.exe cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD ------------------------------------------------------------------------------- Which it is installed on e: instead of c: but there is not a reactor.exe file in the bin directory either. From that information I am thinking I should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as e:\cm3. This is FYI and will let you know if I continue and any other experiences that might be of use. Thank You, Micah -- Ticket URL: CM3 Critical Mass Modula3 Compiler ----- End forwarded message ----- -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 9 16:29:58 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 15:29:58 +0000 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT In-Reply-To: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com> References: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com> Message-ID: Try the .msi instead. Try this maybe, I can't find the link on the web page (maybe lost in the crash?) http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/www/installation-windows.html?rev=1.5;content-type=text%2Fplain I should edit/rewrite those. - The bit about "upgrade is pessmistic" isn't true, as my Russian friend says, it is "realistic". There are several breaking changes that motivate doing it the way upgrade does it. - I should steer people to Python instead of cmd (really). - Mentions of 5.2.6 and 5.5.0 and maybe even building from source should probably go away. Leave this for "more beginners". - I need to update the Visual C++ redistributable link (9.0SP1 vs. 9.0). - The config file path is wrong. It is not config-no-install instead of config. - It alludes to too many options, using various versions, building from source or not, etc. But it seems not too bad. It should be retested. The IDE is not what you would normally call an IDE. It is custom web server. Seems like "apples and oranges", but it does make some sense. In some ways it was ahead of its time -- people talk about "application servers" these days, and run C# in the web server to produce html. Just not clear what sorts of apps are suited to this model. There's no editor -- not a bad idea, since everyone prefers what they already have. There's no debugger -- not a bad idea, not like we have the time to write another. You can use Visual Studio or windbg. There is a bit of a project system. However Modula-3 has among the best text file driven command line build systems around, we just need it to handle walking directories, and have it determine the order. It does provide hyperlinking built source, though personally I just use my editor's "find in files" all the time. Maybe someone else can write up something to replace. http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/www/installation-windows.html?rev=1.5;content-type=text%2Fplain#buildm3current Fixing the .cmd to find the .exes should be trivial. Or just have user run the .exes. Have the .exes set any environment variables they need. - Jay > Date: Tue, 9 Mar 2010 15:25:48 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: [M3devel] Fwd: [CM3] #1082: Windows NT > > some windows problems again, probably only help needed... > > any takers? > > Olaf > > ----- Forwarded message from bugs at elego.de ----- > Date: Tue, 09 Mar 2010 02:55:44 -0000 > From: CM3 > Reply-To: CM3 > Subject: [CM3] #1082: Windows NT > To: @MISSING_DOMAIN > > #1082: Windows NT > -------------------------------------+-------------------------------------- > Reporter: ilikesci@? | Owner: wagner > Type: sw-test | Status: new > Priority: medium | Milestone: > Component: misc | Version: 5.8-RC3 > Severity: non-critical | Keywords: > Relnote: | Org: > Estimatedhours: 0 | Hours: 0 > Billable: 0 | Totalhours: 0 > Internal: 0 | > -------------------------------------+-------------------------------------- > Htr: > Remove msys out of your path. > > > Fix: > > > > Env: > Microsoft Windows Vista > > -------------------------------------+-------------------------------------- > I am new to modula-3 but have tried a few times to get cm3 working on my > MS-Windows machines over the years. This is my new attempt at that. I > downloaded the NT386 files and have them unpacked. I have put cm3 in > e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is > that when I ran cminstall it asked for a tar.exe, gzip.exe, and > msys-1.0.dll. I just happened to have msys installed on my computer and > dropped them in the folder and cminstall seemed to work. I just wanted to > let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and > so I ran it per the suggestion on your website. It asked a few questions > about what browser I wanted to use and such. The browser does pop up on > localhost:3800 but times out. The console spits out: > > Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 > calling start_browser(http://localhost:3800/) > starting TCP service > start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" > http://localhost:3800/ > CM3-IDE is shutting down because start_browser() returned TRUE. > TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept > TCPServer: aborting... > > I am thinking maybe something else might be running on the port and > causing a problem loading page. I copied the startReactor.cmd to the bin > directory and tried it and got the following: > > > ------------------------------------------------------------------------------- > startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by > RCC > > ------------------------------------------------------------------------------- > FATAL ERROR: Unable to find CM3 installation. > CM3_ROOT expected in folder C:\cm3 > CM3_BIN expected in folder C:\cm3\bin > CM3.EXE expected in file C:\cm3\bin\cm3.exe > Reactor.EXE expected in file C:\cm3\bin\reactor.exe > cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD > > ------------------------------------------------------------------------------- > Which it is installed on e: instead of c: but there is not a reactor.exe > file in the bin directory either. From that information I am thinking I > should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as > e:\cm3. > > This is FYI and will let you know if I continue and any other experiences > that might be of use. > > Thank You, > Micah > > -- > Ticket URL: > CM3 > Critical Mass Modula3 Compiler > > > ----- End forwarded message ----- > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 9 16:36:18 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 15:36:18 +0000 Subject: [M3devel] 386?486?586?686?etc.? In-Reply-To: References: , , <20100208035847.GB13743@topoi.pooq.com>, Message-ID: > I'm still running an old 100 MHz Pentium and using it on a daily basis. There was a recent thread on the gcc lists about this. Fedora 11 requires Pentium ("586"). Fedora 12 requires Pentium Pro ("686"). Various distros also use xz/lzma instead of gzip. But OpenBSD sticks with gzip for perf on old machines. Our .deb files use lzma, if the machine building them has the tools. >> Wow. What for? And with Modula-3? What OS? - Jay From: jay.krell at cornell.edu To: hendrik at topoi.pooq.com; m3devel at elegosoft.com Date: Mon, 8 Feb 2010 15:22:21 +0000 Subject: Re: [M3devel] 386?486?586?686?etc.? Wow. What for? And with Modula-3? What OS? I think Pentium will be ok. I think ultimately, if people really need, we should have separate targets. As I've been saying, like: I386_FREEBSD_USERTHREADS, I586_FREEBSD, etc. Esp. to enable easier "release engineering", such as when we do more cross builds, adding new targets will be cheaper. But we'd want some sort of system where if nobody downloads and installs and minimally tests a release, it is in some low grade classification. Certain ones we'd test automatically, like whatever we have available in Hudson. -Jay > Date: Sun, 7 Feb 2010 22:58:48 -0500 > From: hendrik at topoi.pooq.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] 386?486?586?686?etc.? > > On Sun, Feb 07, 2010 at 11:59:11PM +0000, Jay K wrote: > > > > Any opinions/counter-opinions on which processors we should support? > > > > Presumably it doesn't vary per platform. > > > > Like, it wouldn't be Linux/586 and FreeBSD/486 or such. > > > > Unless maybe there is clear data about what the kernels support? > > > > The atomic stuff is pushing things to i586. > > I believe before 486 and possibly 386 worked. > > > > 686 is probably reasonable. > > > > I think it is Pentium II or Pentium Pro or newer, stuff like 15 years old already. > > I'm still running an old 100 MHz Pentium and using it on a daily basis. > > Debian has dropped support for the 386 with, as far as I know, no > complaints. > > -- hendrik. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 9 16:53:25 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 15:53:25 +0000 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT In-Reply-To: References: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com>, Message-ID: Sadly the .msis don't appear on the download page. Perhaps they are there but not indexed? Not that I can tell. There are some here: http://modula3.elegosoft.com/cm3/uploaded-archives/ I'll try them out and make some new ones, from the release branch. Last I tried, and probably currently, there is an unfortunate strict matching requirement between our build and what version of Visual C++ you use. Therefore it behooves us to e.g. provide Visual C++ 8.0 and 9.0 builds. I suggested we double up the Hudson NT386 tasks in that way. I can upload pairs (or more) of .msis, but we are sort of supposed to not take "random developer builds" but use "official package builds". I find the fact that we produce 10+ packages per platform..confusing, overwhelming. I wish there was just one or perhaps two, even though they'd be large. I understand people like to factor things into small pieces, but monolithism has its upsides. - Jay From: jay.krell at cornell.edu To: wagner at elegosoft.com; m3devel at elegosoft.com Date: Tue, 9 Mar 2010 15:29:58 +0000 Subject: Re: [M3devel] Fwd: [CM3] #1082: Windows NT Try the .msi instead. Try this maybe, I can't find the link on the web page (maybe lost in the crash?) http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/www/installation-windows.html?rev=1.5;content-type=text%2Fplain I should edit/rewrite those. - The bit about "upgrade is pessmistic" isn't true, as my Russian friend says, it is "realistic". There are several breaking changes that motivate doing it the way upgrade does it. - I should steer people to Python instead of cmd (really). - Mentions of 5.2.6 and 5.5.0 and maybe even building from source should probably go away. Leave this for "more beginners". - I need to update the Visual C++ redistributable link (9.0SP1 vs. 9.0). - The config file path is wrong. It is not config-no-install instead of config. - It alludes to too many options, using various versions, building from source or not, etc. But it seems not too bad. It should be retested. The IDE is not what you would normally call an IDE. It is custom web server. Seems like "apples and oranges", but it does make some sense. In some ways it was ahead of its time -- people talk about "application servers" these days, and run C# in the web server to produce html. Just not clear what sorts of apps are suited to this model. There's no editor -- not a bad idea, since everyone prefers what they already have. There's no debugger -- not a bad idea, not like we have the time to write another. You can use Visual Studio or windbg. There is a bit of a project system. However Modula-3 has among the best text file driven command line build systems around, we just need it to handle walking directories, and have it determine the order. It does provide hyperlinking built source, though personally I just use my editor's "find in files" all the time. Maybe someone else can write up something to replace. http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/www/installation-windows.html?rev=1.5;content-type=text%2Fplain#buildm3current Fixing the .cmd to find the .exes should be trivial. Or just have user run the .exes. Have the .exes set any environment variables they need. - Jay > Date: Tue, 9 Mar 2010 15:25:48 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: [M3devel] Fwd: [CM3] #1082: Windows NT > > some windows problems again, probably only help needed... > > any takers? > > Olaf > > ----- Forwarded message from bugs at elego.de ----- > Date: Tue, 09 Mar 2010 02:55:44 -0000 > From: CM3 > Reply-To: CM3 > Subject: [CM3] #1082: Windows NT > To: @MISSING_DOMAIN > > #1082: Windows NT > -------------------------------------+-------------------------------------- > Reporter: ilikesci@? | Owner: wagner > Type: sw-test | Status: new > Priority: medium | Milestone: > Component: misc | Version: 5.8-RC3 > Severity: non-critical | Keywords: > Relnote: | Org: > Estimatedhours: 0 | Hours: 0 > Billable: 0 | Totalhours: 0 > Internal: 0 | > -------------------------------------+-------------------------------------- > Htr: > Remove msys out of your path. > > > Fix: > > > > Env: > Microsoft Windows Vista > > -------------------------------------+-------------------------------------- > I am new to modula-3 but have tried a few times to get cm3 working on my > MS-Windows machines over the years. This is my new attempt at that. I > downloaded the NT386 files and have them unpacked. I have put cm3 in > e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is > that when I ran cminstall it asked for a tar.exe, gzip.exe, and > msys-1.0.dll. I just happened to have msys installed on my computer and > dropped them in the folder and cminstall seemed to work. I just wanted to > let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and > so I ran it per the suggestion on your website. It asked a few questions > about what browser I wanted to use and such. The browser does pop up on > localhost:3800 but times out. The console spits out: > > Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 > calling start_browser(http://localhost:3800/) > starting TCP service > start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" > http://localhost:3800/ > CM3-IDE is shutting down because start_browser() returned TRUE. > TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept > TCPServer: aborting... > > I am thinking maybe something else might be running on the port and > causing a problem loading page. I copied the startReactor.cmd to the bin > directory and tried it and got the following: > > > ------------------------------------------------------------------------------- > startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by > RCC > > ------------------------------------------------------------------------------- > FATAL ERROR: Unable to find CM3 installation. > CM3_ROOT expected in folder C:\cm3 > CM3_BIN expected in folder C:\cm3\bin > CM3.EXE expected in file C:\cm3\bin\cm3.exe > Reactor.EXE expected in file C:\cm3\bin\reactor.exe > cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD > > ------------------------------------------------------------------------------- > Which it is installed on e: instead of c: but there is not a reactor.exe > file in the bin directory either. From that information I am thinking I > should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as > e:\cm3. > > This is FYI and will let you know if I continue and any other experiences > that might be of use. > > Thank You, > Micah > > -- > Ticket URL: > CM3 > Critical Mass Modula3 Compiler > > > ----- End forwarded message ----- > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 9 17:46:52 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 9 Mar 2010 16:46:52 +0000 Subject: [M3devel] PPC64_DARWIN Message-ID: Fyi, PPC64_DARWIN mostly works. "Ship" has a strong tendency to hang. Whenever I've looked, the stack has no symbols. I tried growing jmpbuf, using default stack, no luck yet. Will look more "later". Anyone running a Mac/G5? It's a big endian 64bit platform, if that is interesting. :) (SPARC64_SOLARIS will be too.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Tue Mar 9 15:43:44 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Tue, 9 Mar 2010 09:43:44 -0500 Subject: [M3devel] 386?486?586?686?etc.? In-Reply-To: References: Message-ID: <20100309144344.GC12734@topoi.pooq.com> On Tue, Mar 09, 2010 at 03:36:18PM +0000, Jay K wrote: > > > I'm still running an old 100 MHz Pentium and using it on a daily basis. > >> Wow. What for? And with Modula-3? What OS? It's my most reliable machine. The kind of reliable that's been running for something like two decades and shows signs of running for two more if I can continue to get secure OS support for it. My newer machines keep burning up graphics cards and the like. It is running Debian Lenny at the moment; a few months ago it was on Debian Etch. It's the boundary between my LAN and the internet. I last used Modula 3 on it a few years ago, throwing together an experimental type-checked language interpreter to test some ideas about language design. I'm not running Modula 3 on it right now, but since Modula 3 is still my systems language of choice, I'd hesitate to say I won't be doing it again. - hendrik > > > - Jay > > > > > > From: jay.krell at cornell.edu > To: hendrik at topoi.pooq.com; m3devel at elegosoft.com > Date: Mon, 8 Feb 2010 15:22:21 +0000 > Subject: Re: [M3devel] 386?486?586?686?etc.? > > > > Wow. What for? And with Modula-3? What OS? > I think Pentium will be ok. > I think ultimately, if people really need, we should have separate targets. > As I've been saying, like: I386_FREEBSD_USERTHREADS, I586_FREEBSD, etc. > Esp. to enable easier "release engineering", such as when we do more cross builds, > adding new targets will be cheaper. But we'd want some sort of system > where if nobody downloads and installs and minimally tests a release, it is > in some low grade classification. > Certain ones we'd test automatically, like whatever we have available in Hudson. > > -Jay > > > Date: Sun, 7 Feb 2010 22:58:48 -0500 > > From: hendrik at topoi.pooq.com > > To: m3devel at elegosoft.com > > Subject: Re: [M3devel] 386?486?586?686?etc.? > > > > On Sun, Feb 07, 2010 at 11:59:11PM +0000, Jay K wrote: > > > > > > Any opinions/counter-opinions on which processors we should support? > > > > > > Presumably it doesn't vary per platform. > > > > > > Like, it wouldn't be Linux/586 and FreeBSD/486 or such. > > > > > > Unless maybe there is clear data about what the kernels support? > > > > > > The atomic stuff is pushing things to i586. > > > I believe before 486 and possibly 386 worked. > > > > > > 686 is probably reasonable. > > > > > > I think it is Pentium II or Pentium Pro or newer, stuff like 15 years old already. > > > > I'm still running an old 100 MHz Pentium and using it on a daily basis. > > > > Debian has dropped support for the 386 with, as far as I know, no > > complaints. > > > > -- hendrik. > From rcolebur at SCIRES.COM Wed Mar 10 03:10:57 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Tue, 9 Mar 2010 21:10:57 -0500 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT In-Reply-To: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com> References: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com> Message-ID: Olaf: The "startReactor.CMD" file is obsolete and no longer in the repository. The new files are in "scripts\install\windows" and should be copied to the "cm3\bin" folder on a windows installation. "cm3StartIDE.CMD" replaces "startReactor.CMD". This file makes use of "cm3CommandShell.CMD". It appears that the problem reporter has his installation rooted at E:\cm3 rather than C:\cm3. So, he will need to set an environment variable "set CM3_ROOT=E:\cm3" to let these scripts know where to find the cm3 installation. Alternately, cm3CommandShell can be invoked with the arguments "Root E:\cm3" to tell it where to find the installation. Regards, Randy -----Original Message----- From: Olaf Wagner [mailto:wagner at elegosoft.com] Sent: Tuesday, March 09, 2010 9:26 AM To: m3devel Subject: [M3devel] Fwd: [CM3] #1082: Windows NT some windows problems again, probably only help needed... any takers? Olaf ----- Forwarded message from bugs at elego.de ----- Date: Tue, 09 Mar 2010 02:55:44 -0000 From: CM3 Reply-To: CM3 Subject: [CM3] #1082: Windows NT To: @MISSING_DOMAIN #1082: Windows NT -------------------------------------+-------------------------------------- Reporter: ilikesci@? | Owner: wagner Type: sw-test | Status: new Priority: medium | Milestone: Component: misc | Version: 5.8-RC3 Severity: non-critical | Keywords: Relnote: | Org: Estimatedhours: 0 | Hours: 0 Billable: 0 | Totalhours: 0 Internal: 0 | -------------------------------------+-------------------------------------- Htr: Remove msys out of your path. Fix: Env: Microsoft Windows Vista -------------------------------------+-------------------------------------- I am new to modula-3 but have tried a few times to get cm3 working on my MS-Windows machines over the years. This is my new attempt at that. I downloaded the NT386 files and have them unpacked. I have put cm3 in e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is that when I ran cminstall it asked for a tar.exe, gzip.exe, and msys-1.0.dll. I just happened to have msys installed on my computer and dropped them in the folder and cminstall seemed to work. I just wanted to let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and so I ran it per the suggestion on your website. It asked a few questions about what browser I wanted to use and such. The browser does pop up on localhost:3800 but times out. The console spits out: Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 calling start_browser(http://localhost:3800/) starting TCP service start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" http://localhost:3800/ CM3-IDE is shutting down because start_browser() returned TRUE. TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept TCPServer: aborting... I am thinking maybe something else might be running on the port and causing a problem loading page. I copied the startReactor.cmd to the bin directory and tried it and got the following: ------------------------------------------------------------------------------- startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by RCC ------------------------------------------------------------------------------- FATAL ERROR: Unable to find CM3 installation. CM3_ROOT expected in folder C:\cm3 CM3_BIN expected in folder C:\cm3\bin CM3.EXE expected in file C:\cm3\bin\cm3.exe Reactor.EXE expected in file C:\cm3\bin\reactor.exe cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD ------------------------------------------------------------------------------- Which it is installed on e: instead of c: but there is not a reactor.exe file in the bin directory either. From that information I am thinking I should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as e:\cm3. This is FYI and will let you know if I continue and any other experiences that might be of use. Thank You, Micah -- Ticket URL: CM3 Critical Mass Modula3 Compiler ----- End forwarded message ----- -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. From jay.krell at cornell.edu Wed Mar 10 07:52:42 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 10 Mar 2010 06:52:42 +0000 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT In-Reply-To: References: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com>, Message-ID: %~dp0 is a cmd file's directory. If cmd and exe are next to each other, they can find each other. As well, what does the cmd do that it can't be done in the exe? - Jay (phone) > From: rcolebur at SCIRES.COM > To: wagner at elegosoft.com; m3devel at elegosoft.com > Date: Tue, 9 Mar 2010 21:10:57 -0500 > Subject: Re: [M3devel] Fwd: [CM3] #1082: Windows NT > > Olaf: > > The "startReactor.CMD" file is obsolete and no longer in the repository. > > The new files are in "scripts\install\windows" and should be copied to the "cm3\bin" folder on a windows installation. > > "cm3StartIDE.CMD" replaces "startReactor.CMD". This file makes use of "cm3CommandShell.CMD". > > It appears that the problem reporter has his installation rooted at E:\cm3 rather than C:\cm3. So, he will need to set an environment variable "set CM3_ROOT=E:\cm3" to let these scripts know where to find the cm3 installation. Alternately, cm3CommandShell can be invoked with the arguments "Root E:\cm3" to tell it where to find the installation. > > Regards, > Randy > > -----Original Message----- > From: Olaf Wagner [mailto:wagner at elegosoft.com] > Sent: Tuesday, March 09, 2010 9:26 AM > To: m3devel > Subject: [M3devel] Fwd: [CM3] #1082: Windows NT > > some windows problems again, probably only help needed... > > any takers? > > Olaf > > ----- Forwarded message from bugs at elego.de ----- > Date: Tue, 09 Mar 2010 02:55:44 -0000 > From: CM3 > Reply-To: CM3 > Subject: [CM3] #1082: Windows NT > To: @MISSING_DOMAIN > > #1082: Windows NT > -------------------------------------+-------------------------------------- > Reporter: ilikesci@? | Owner: wagner > Type: sw-test | Status: new > Priority: medium | Milestone: > Component: misc | Version: 5.8-RC3 > Severity: non-critical | Keywords: > Relnote: | Org: > Estimatedhours: 0 | Hours: 0 > Billable: 0 | Totalhours: 0 > Internal: 0 | > -------------------------------------+-------------------------------------- > Htr: > Remove msys out of your path. > > > Fix: > > > > Env: > Microsoft Windows Vista > > -------------------------------------+-------------------------------------- > I am new to modula-3 but have tried a few times to get cm3 working on my > MS-Windows machines over the years. This is my new attempt at that. I > downloaded the NT386 files and have them unpacked. I have put cm3 in > e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is > that when I ran cminstall it asked for a tar.exe, gzip.exe, and > msys-1.0.dll. I just happened to have msys installed on my computer and > dropped them in the folder and cminstall seemed to work. I just wanted to > let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and > so I ran it per the suggestion on your website. It asked a few questions > about what browser I wanted to use and such. The browser does pop up on > localhost:3800 but times out. The console spits out: > > Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 > calling start_browser(http://localhost:3800/) > starting TCP service > start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" > http://localhost:3800/ > CM3-IDE is shutting down because start_browser() returned TRUE. > TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept > TCPServer: aborting... > > I am thinking maybe something else might be running on the port and > causing a problem loading page. I copied the startReactor.cmd to the bin > directory and tried it and got the following: > > > ------------------------------------------------------------------------------- > startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by > RCC > > ------------------------------------------------------------------------------- > FATAL ERROR: Unable to find CM3 installation. > CM3_ROOT expected in folder C:\cm3 > CM3_BIN expected in folder C:\cm3\bin > CM3.EXE expected in file C:\cm3\bin\cm3.exe > Reactor.EXE expected in file C:\cm3\bin\reactor.exe > cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD > > ------------------------------------------------------------------------------- > Which it is installed on e: instead of c: but there is not a reactor.exe > file in the bin directory either. From that information I am thinking I > should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as > e:\cm3. > > This is FYI and will let you know if I continue and any other experiences > that might be of use. > > Thank You, > Micah > > -- > Ticket URL: > CM3 > Critical Mass Modula3 Compiler > > > ----- End forwarded message ----- > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > > > CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. > > EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Wed Mar 10 14:10:41 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Wed, 10 Mar 2010 14:10:41 +0100 Subject: [M3devel] [CM3] #1082: Windows NT Message-ID: <20100310141041.t3hp8s034ss0cggc@mail.elegosoft.com> Have you read the answers on the m3devel list to which I forwarded your report? If not, you will find them in the archives at https://mail.elegosoft.com/pipermail/m3devel/2010-March/thread.html Hope that helps, @Jay/Randy, you should add your information to the trac ticket, too: https://projects.elego.de/cm3/ticket/1082 Sorry that I cannot do much myself currently, Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From rcolebur at SCIRES.COM Wed Mar 10 16:35:29 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Wed, 10 Mar 2010 10:35:29 -0500 Subject: [M3devel] Fwd: [CM3] #1082: Windows NT In-Reply-To: References: <20100309152548.hk9lq6fm88s0gk40@mail.elegosoft.com>, Message-ID: Jay, the CMD files are conveniences I've found useful under Windows. I can supply some .REG files that will allow you to integrate these into the Explorer shell so that you can start a cm3 command shell in any folder and you can start CM3IDE. These CMD files ensure the environment is set up properly, including Visual C++. I understand the use of %~dp0 , but I already have an optional command line argument to deal with different locations for cm3 installation. Nevertheless, I can build in some more search capability in the cmd files if desired. Perhaps using %~dp0 would be a good tactic; I'll look into adding it. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, March 10, 2010 1:53 AM To: Coleburn, Randy; Olaf Wagner; m3devel Subject: RE: [M3devel] Fwd: [CM3] #1082: Windows NT %~dp0 is a cmd file's directory. If cmd and exe are next to each other, they can find each other. As well, what does the cmd do that it can't be done in the exe? - Jay (phone) > From: rcolebur at SCIRES.COM > To: wagner at elegosoft.com; m3devel at elegosoft.com > Date: Tue, 9 Mar 2010 21:10:57 -0500 > Subject: Re: [M3devel] Fwd: [CM3] #1082: Windows NT > > Olaf: > > The "startReactor.CMD" file is obsolete and no longer in the repository. > > The new files are in "scripts\install\windows" and should be copied to the "cm3\bin" folder on a windows installation. > > "cm3StartIDE.CMD" replaces "startReactor.CMD". This file makes use of "cm3CommandShell.CMD". > > It appears that the problem reporter has his installation rooted at E:\cm3 rather than C:\cm3. So, he will need to set an environment variable "set CM3_ROOT=E:\cm3" to let these scripts know where to find the cm3 installation. Alternately, cm3CommandShell can be invoked with the arguments "Root E:\cm3" to tell it where to find the installation. > > Regards, > Randy > > -----Original Message----- > From: Olaf Wagner [mailto:wagner at elegosoft.com] > Sent: Tuesday, March 09, 2010 9:26 AM > To: m3devel > Subject: [M3devel] Fwd: [CM3] #1082: Windows NT > > some windows problems again, probably only help needed... > > any takers? > > Olaf > > ----- Forwarded message from bugs at elego.de ----- > Date: Tue, 09 Mar 2010 02:55:44 -0000 > From: CM3 > Reply-To: CM3 > Subject: [CM3] #1082: Windows NT > To: @MISSING_DOMAIN > > #1082: Windows NT > -------------------------------------+-------------------------------------- > Reporter: ilikesci at ... | Owner: wagner > Type: sw-test | Status: new > Priority: medium | Milestone: > Component: misc | Version: 5.8-RC3 > Severity: non-critical | Keywords: > Relnote: | Org: > Estimatedhours: 0 | Hours: 0 > Billable: 0 | Totalhours: 0 > Internal: 0 | > -------------------------------------+-------------------------------------- > Htr: > Remove msys out of your path. > > > Fix: > > > > Env: > Microsoft Windows Vista > > -------------------------------------+-------------------------------------- > I am new to modula-3 but have tried a few times to get cm3 working on my > MS-Windows machines over the years. This is my new attempt at that. I > downloaded the NT386 files and have them unpacked. I have put cm3 in > e:\cm3 and have e:\cm3\bin in my path. The first problem I came across is > that when I ran cminstall it asked for a tar.exe, gzip.exe, and > msys-1.0.dll. I just happened to have msys installed on my computer and > dropped them in the folder and cminstall seemed to work. I just wanted to > let you know. Second, e:\cm3\bin\cm3ide.exe was in the bin directory and > so I ran it per the suggestion on your website. It asked a few questions > about what browser I wanted to use and such. The browser does pop up on > localhost:3800 but times out. The console spits out: > > Recovering user state from E:\cm3\bin\CM3_IDE.cfg1 > calling start_browser(http://localhost:3800/) > starting TCP service > start /wait "C:\Program Files\Mozilla Firefox\firefox.exe" > http://localhost:3800/ > CM3-IDE is shutting down because start_browser() returned TRUE. > TCPServer: IP.Error: TCP.Unexpected *** 10093 *** TCP.Accept > TCPServer: aborting... > > I am thinking maybe something else might be running on the port and > causing a problem loading page. I copied the startReactor.cmd to the bin > directory and tried it and got the following: > > > ------------------------------------------------------------------------------- > startReactor.CMD, written by R.C.Coleburn 08/13/2003, v1.13 08/29/2003 by > RCC > > ------------------------------------------------------------------------------- > FATAL ERROR: Unable to find CM3 installation. > CM3_ROOT expected in folder C:\cm3 > CM3_BIN expected in folder C:\cm3\bin > CM3.EXE expected in file C:\cm3\bin\cm3.exe > Reactor.EXE expected in file C:\cm3\bin\reactor.exe > cm3SetupCmdEnv.CMD expected in file C:\cm3\bin\cm3SetupCmdEnv.CMD > > ------------------------------------------------------------------------------- > Which it is installed on e: instead of c: but there is not a reactor.exe > file in the bin directory either. From that information I am thinking I > should define CM3_ROOT and CM3_BIN in my environment? I put CM3_HOME as > e:\cm3. > > This is FYI and will let you know if I continue and any other experiences > that might be of use. > > Thank You, > Micah > > -- > Ticket URL: > CM3 > Critical Mass Modula3 Compiler > > > ----- End forwarded message ----- > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > > > CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. > > EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 10 16:56:32 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 10 Mar 2010 15:56:32 +0000 Subject: [M3devel] NT386 64bit LONGINT sub range checks oops Message-ID: Just a note that subrange checking of 64bit LONGINT is broken on NT386. I just noticed. 64bit LONGINT should work to a very large extent though. This is the only problem I know of. I'll look into it soon. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 10 17:09:46 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 10 Mar 2010 16:09:46 +0000 Subject: [M3devel] NT386 64bit LONGINT sub range checks oops In-Reply-To: References: Message-ID: Hm. I'm just not sure actually. I'll have to test it. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Wed, 10 Mar 2010 15:56:32 +0000 Subject: [M3devel] NT386 64bit LONGINT sub range checks oops Just a note that subrange checking of 64bit LONGINT is broken on NT386. I just noticed. 64bit LONGINT should work to a very large extent though. This is the only problem I know of. I'll look into it soon. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 11 15:45:21 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 11 Mar 2010 14:45:21 +0000 Subject: [M3devel] interface long, inlining large integer code, etc. In-Reply-To: <60FB82E7-B3B3-45ED-973B-191F7992A4E2@cs.purdue.edu> References: <20100308122636.83CA12474001@birch.elegosoft.com>, , , , , , , , , , , , , <5ECDB225-60F1-4094-96EA-B7BB0EBE340A@cs.purdue.edu>, , <60FB82E7-B3B3-45ED-973B-191F7992A4E2@cs.purdue.edu> Message-ID: I was thinking a bit more about this. Basically it can already be done -- we know the name of the function are compiling. If the function is Long__Extract, Long__Times, etc., we can inline. If it isn't, we can assume the existance and interface of such functions, and call them. Instead of calling anything in hand.c. Maybe even give them custom calling conventions. The front end knows stuff about interface Word/Long. It seems reasonable for m3back to also. The open question however is that there is no place to "hang" inlined versions of signed operations, what you might call Integer__Extract, Integer__Divide, LongInt__Div, LongInt__Times, LongInt__Mod, etc. Maybe they should be in C:\dev2\cm3.2\m3-libs\m3core\src\types? Similarly open question would be the set functions. RTHooks.SetLT? RTSet.LT? C:\dev2\cm3.2\m3-libs\m3core\src\types\set? C:\dev2\cm3.2\m3-libs\m3core\src\types\bigset? (as opposed to word-sized set, besides that eq/ne are inlined by the frontend for "medium" sizes) Is it clear what my questions are? Let me try again. Consider Long.Rotate. Assume it takes a bunch of code. m3back could notice it is compiling Long__Rotate, and generate the code inline. Otherwise it can "know" such a function exists and generate a call to it. This is a reasonable seeming model that I just came up with. That does double the paths in m3back, granted. For several such m3cg operations, there is already a natural function to treat this way -- stuff in interface Long. However, for a few operations, there is not already a place -- signed operations and operations on large sets. As well, there is a presumed question/answer/non-answer: everything works today. But should it be changed? Is it better to reduce/eliminate hand.c and instead teach the Modula-3 code in m3back how to generate assembly? "Better" how? More efficient? Maybe. The factor here is that hand.c may or may not be optimized, but m3back operates in a mode where it always optimizes to some extent. It generally generates code that is significantly better than unoptimized C, and significantly worse than optimized C, depending -- it does pretty well, but it never inlines, but it only does a certain small class of smart things. It keeps values in registers somewhat, it doesn't eliminate dead stores, doesn't unroll loops. Another factor is the principle of having C code or not. We vehemently agree that C code is preferred in m3core/unix. But otherwise many people here kind of dislike or discourage C on principle/philosophical grounds. I'm not sure I agree, however by agreeing, I introduce an interesting technical challenge. :) Again, must we really call it interface long? Does that really "sound like": INTEGER is to Word as LONGINT is to x? LongWord maybe? - Jay > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > CC: m3commit at elegosoft.com > Subject: RE: [M3commit] CVS Update: cm3 > Date: Thu, 11 Mar 2010 04:10:40 +0000 > > > Long.LeftShift is not a great example. Sorry. > Long.Shift, Long.Rotate, Long.Times, LongDivide. > Those are or "could be" large. Currently the later > three all call functions and are therefore short. > > > Integer.Multiple, Divide, Remainder, similarly uncertain. > They call functions, the functions aren't short. > > > I was also thinking, some of these functions, esp. > 64bit shift/rotate could be shorter as loops. > But I think not worthwhile. > > > To be clear, even though "parse.c" "never" generates > function calls (except for set operations), the gcc > backend itself often does. The functions are in libgcc. > > > You know..I find it hard to decide about compiler generated > function calls. Definitely for size optimization they > can be a win. But I just somehow vaguely don't like them. > In many cases, the decision is not difficult. set_singleton > and set_member are good examples. They could be inlined > and be very small. But Shift, Rotate, Multiply, Divide, > it gets less clear. > > > But no matter how large an inline form, I think > when interface word/long maps directly to these operations, > they should be inlined. > > > And then if there are places that don't inline, they should > call the "instantiations" in interface word/long. > > > And then, furthermore, we should consider an interface > for "Integer.Divide/Multiple/Remainder"? > A place to park the possibly large code? > > > - Jay > > > ---------------------------------------- > > From: hosking at cs.purdue.edu > > Date: Wed, 10 Mar 2010 21:01:54 -0500 > > To: jay.krell at cornell.edu > > CC: m3commit at elegosoft.com > > Subject: Re: [M3commit] CVS Update: cm3 > > > > I am confused. The intention is that Long.LeftShift is always inlined wherever it is called. Is the code for it really that hairy? > > > > On 10 Mar 2010, at 20:30, Jay K wrote: > > > >> > >> If there exists an inline expansion, and it is large, but there is a function whose job it is to hold the expansion, whether it is inline or not, and the backend was informed of this function's name, and whether or not it was currently producing it, the backend could generate the inline expansion for Long__LeftShift, but otherwise generate calls to it. > >> > >> > >> That is, e.g. m3cg.left_shift(type := int64|word64) could chose to either call Long__LeftShift or generate the body of Long__LeftShift. Not based on some "is inlining profitable heuristic", but specifically if told it is generating the function Long__LeftShift or not. > >> > >> > >> That is, there's no point in having a C function m3_left_shift64 or somesuch, and having Long__LeftShift call it. Instead a backend should generate the body of Long__LeftShift inline when told it is generating that function, vs. generating a call to that function when it is otherwise asked to do a 64bit leftshift if it decides that inlining it every time is too large. > >> > >> Granted at least two things: > >> Given 64bit target, Long__LeftShift vs. Word__LeftShift is ambiguous. > >> And I'm inlined to just always inline anyway. > >> > >> > >> (I still don't like the term "Long" here. It doesn't convey unsigned.) > >> > >> > >> - Jay > >> > >> > >> > >> ________________________________ > >>> Subject: Re: [M3commit] CVS Update: cm3 > >>> From: hosking at cs.purdue.edu > >>> Date: Wed, 10 Mar 2010 19:57:17 -0500 > >>> CC: m3commit at elegosoft.com > >>> To: jay.krell at cornell.edu > >>> > >>> > >>> > >>> Yes, someone can pass the function as a parameter. > >>> > >>> I don't understand the rest of what you are saying. > >>> > >>> > >>> Antony Hosking | Associate Professor | Computer Science | Purdue University > >>> 305 N. University Street | West Lafayette | IN 47907 | USA > >>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > >>> > >>> > >>> > >>> > >>> > >>> > >>> On 10 Mar 2010, at 17:27, Jay K wrote: > >>> > >>> > >>> Hey, should we maybe have a flag that indicates we are producing a function that is meant to provide and only provide exactly the functionality of the "one" m3cg call? > >>> You know, that'd be a great hint to: > >>> > >>> - definitely definitely definitely definitely inline > >>> - if we know this thing existed, we could use it instead of a .c file. > >>> - along with the function's name for the invocations that aren't the function? > >>> > >>> > >>> Why provide the function anyway? In case someone takes the address? > >>> > >>> > >>> > >>> - Jay > >>> > >>> > >>> ________________________________ > >>> From: jay.krell at cornell.edu > >>> To: hosking at cs.purdue.edu > >>> Date: Mon, 8 Mar 2010 16:04:45 +0000 > >>> CC: m3commit at elegosoft.com > >>> Subject: Re: [M3commit] CVS Update: cm3 > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> Fair enough. > >>> > >>> > >>> > >>> - Jay > >>> > >>> > >>> > >>> ________________________________ > >>> > >>> From: hosking at cs.purdue.edu > >>> Date: Mon, 8 Mar 2010 10:29:00 -0500 > >>> To: jay.krell at cornell.edu > >>> CC: m3commit at elegosoft.com > >>> Subject: Re: [M3commit] CVS Update: cm3 > >>> > >>> > >>> > >>> > >>> You can't call the support function Long__Extract because we already have a Long__Extract in m3core/src/word. It is the library representative of the front-end compiler-generated (inlined) Long.Extract. > >>> > >>> > >>> > >>> On 8 Mar 2010, at 07:30, Jay K wrote: > >>> > >>> > >>> > >>> diff attached > >>> > >>> > >>> If folks really want to use tables or function calls here, let me know. > >>> > >>> > >>> The data was historically problematic, though we worked out the problems. > >>> - It was initialized at runtime which has a race condition, fixed years ago. > >>> - It can't be "exported" on NT386 (which is the only platform that uses it) and must be statically linked. > >>> > >>> The data is still used by Word.Insert and Long.Insert is still a function, but those > >>> are my next targets. > >>> > >>> > >>> This is a case where the user does write a function call. Word.Extract or Long.Extract. > >>> (So the function should have been called Long__Extract.) > >>> > >>> > >>> - Jay > >>> > >>> Date: Mon, 8 Mar 2010 13:26:36 +0000 > >>> To: m3commit at elegosoft.com > >>> From: jkrell at elego.de > >>> Subject: [M3commit] CVS Update: cm3 > >>> > >>> CVSROOT: /usr/cvs > >>> Changes by: jkrell at birch. 10/03/08 13:26:36 > >>> > >>> Modified files: > >>> cm3/m3-sys/m3back/src/: M3x86.m3 Stackx86.i3 Stackx86.m3 > >>> > >>> Log message: > >>> "rewrite" extract to not use tables and always be inlined for 64bit > >>> > >>> equivalent C code: > >>> > >>> UT __stdcall extract(UT x, uint32 offset, uint32 count) > >>> { > >>> x>>= offset; > >>> x &= ~((~(UT)0) << count); > >>> return x; > >>> } > >>> > >>> extract32 > >>> 00000729: 8B4DEC MOV ECX tv.126[_m] > >>> 0000072C: 8B5DF4 MOV EBX tv.124[_a32] > >>> 0000072F: D3EB SHR EBX > >>> 00000731: 8BD1 MOV EDX ECX This is pointless and I'll try to remove. > >>> 00000733: 8B4DE4 MOV ECX tv.128[_n] > >>> 00000736: BEFFFFFFFF MOV ESI $4294967295 I'll look for smaller form of this. or esi -1? > >>> 0000073B: D3E6 SHL ESI > >>> 0000073D: F7D6 NOT ESI > >>> 0000073F: 23DE AND EBX ESI > >>> > >>> extract64 > >>> 000008E4: 8B4DD8 MOV ECX tv.134[_m] > >>> 000008E7: 8B5DE8 MOV EBX tv.131[_a64] > >>> 000008EA: 8B55EC MOV EDX tv.131[_a64]+4 > >>> 000008ED: 0FADD3 SHRD EBX EDX ECX > >>> 000008F0: D3EA SHR EDX > >>> 000008F2: F6C120 TEST ECX $32 > >>> 000008F5: 7400 JE rel8 L.107 > >>> 000008F7: 8BDA MOV EBX EDX > >>> 000008F9: 33D2 XOR EDX EDX > >>> set_label L.107 > >>> 000008FB: 8BF1 MOV ESI ECX This is pointless and I'll try to remove. > >>> 000008FD: 8B4DD0 MOV ECX tv.136[_n] > >>> 00000900: BFFFFFFFFF MOV EDI $4294967295 I'll look for smaller form of this. or edi -1? > >>> 00000905: B8FFFFFFFF MOV EAX $4294967295 I'll look for smaller form of this. (heck, at least mov edi, eax) > >>> 0000090A: 0FA5F8 SHLD EAX EDI ECX > >>> 0000090D: D3E7 SHL EDI > >>> 0000090F: F6C120 TEST ECX $32 > >>> 00000912: 7400 JE rel8 L.108 > >>> 00000914: 8BC7 MOV EAX EDI > >>> 00000916: 33FF XOR EDI EDI > >>> set_label L.108 > >>> 00000918: F7D7 NOT EDI > >>> 0000091A: F7D0 NOT EAX > >>> 0000091C: 23DF AND EBX EDI > >>> 0000091E: 23D0 AND EDX EAX > >>> > >>> having n or m and n (or just m? I think so.) be constant leads to better code > >>> > >>> some other small cleanup, like avoiding calling find twice, > >>> I don't see why it was that way > >>> > >>> <1.txt> > >>> > >>> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 11 16:40:34 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 11 Mar 2010 15:40:34 +0000 Subject: [M3devel] Target.EOL Message-ID: I strongly suspect that Target.EOL can go away and just use Wr.EOL instead. Or even just \n. Cross builds are fairly rare, esp. cross between NT and Posix, and very many tools treat \n, \r\n, and perhaps \r the same, so even crossing NT <=> Posix doesn't likely matter. ie. it works, assuming you have a cross m3cg (ie: mingwin/cygwin hosted for NT hosts) gcc, Visual C++ compiler, m3front, all treat \r and \r\n the same. (witness all our *.h, *.c, *.i3, *.m3 files use \r\n) I think we have some tools that don't understand \r\n, but in my opinion that is a bug. All three formats should be treated the same. I know notepad doesn't display \n well, and Sun cc might not like \r\n. I know some compiler I used recently was picky, but "many" compilers are not. cmd might be ok with \n. Python calls it "universal newlines", treating all three formats the same. C:\dev2\cm3.2\m3-sys\cm3\src\Utils.m3(190):PROCEDURE CopyText (old, new: TEXT) = C:\dev2\cm3.2\m3-sys\m3middle\src\M3File.m3(61):PROCEDURE CopyText (src, dest: TEXT; eol: TEXT) RAISES {OSError.E} = probably leave alone, except that they don't treat \r correctly if you consider the historical Apple behavior. There are three formats, not two. Anyway, it's not a big deal. I think my problem here is solved by the change to m3x86.m3 I already made. I'll put Target.EOL back on my machine. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 11 18:31:42 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 11 Mar 2010 12:31:42 -0500 Subject: [M3devel] Target.EOL In-Reply-To: References: Message-ID: <6A2F1518-1C9C-4666-BC3E-B6B8D397A9EA@cs.purdue.edu> Sounds reasonable. On 11 Mar 2010, at 10:40, Jay K wrote: > I strongly suspect that Target.EOL can go away and just use Wr.EOL instead. > Or even just \n. > > > Cross builds are fairly rare, esp. cross between NT and Posix, and very > many tools treat \n, \r\n, and perhaps \r the same, so even > crossing NT <=> Posix doesn't likely matter. > ie. it works, assuming you have a cross m3cg (ie: mingwin/cygwin hosted for NT hosts) > > > gcc, Visual C++ compiler, m3front, all treat \r and \r\n the same. > (witness all our *.h, *.c, *.i3, *.m3 files use \r\n) > I think we have some tools that don't understand \r\n, but in my opinion that is a bug. > All three formats should be treated the same. > > > I know notepad doesn't display \n well, and Sun cc might not like \r\n. > I know some compiler I used recently was picky, but "many" compilers are not. > cmd might be ok with \n. > Python calls it "universal newlines", treating all three formats the same. > > > C:\dev2\cm3.2\m3-sys\cm3\src\Utils.m3(190):PROCEDURE CopyText (old, new: TEXT) = > C:\dev2\cm3.2\m3-sys\m3middle\src\M3File.m3(61):PROCEDURE CopyText (src, dest: TEXT; eol: TEXT) RAISES {OSError.E} = > > > probably leave alone, except that they don't treat \r correctly if you consider the historical Apple behavior. > There are three formats, not two. > > > Anyway, it's not a big deal. I think my problem here is solved by the change to m3x86.m3 I already made. > I'll put Target.EOL back on my machine. > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 11 19:03:43 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 11 Mar 2010 13:03:43 -0500 Subject: [M3devel] interface long, inlining large integer code, etc. In-Reply-To: References: <20100308122636.83CA12474001@birch.elegosoft.com>, , , , , , , , , , , , , <5ECDB225-60F1-4094-96EA-B7BB0EBE340A@cs.purdue.edu>, , <60FB82E7-B3B3-45ED-973B-191F7992A4E2@cs.purdue.edu> Message-ID: <8536088B-FFE4-43CA-AC63-442D66AE72A0@cs.purdue.edu> I'm going to resist this one. It's not clear that it matters terribly much whether these utility routines are inlined or not. If it is shown to be a performance bottleneck then perhaps. But I suspect that there are very few programs that pass the procedures of the Word/Long interface around by value. On 11 Mar 2010, at 09:45, Jay K wrote: > I was thinking a bit more about this. > Basically it can already be done -- we know the name of the function are compiling. > If the function is Long__Extract, Long__Times, etc., we can inline. > If it isn't, we can assume the existance and interface of such functions, and call them. > Instead of calling anything in hand.c. > Maybe even give them custom calling conventions. > The front end knows stuff about interface Word/Long. It seems reasonable for m3back to also. > > > The open question however is that there is no place to "hang" inlined versions of signed operations, what you might call Integer__Extract, Integer__Divide, LongInt__Div, LongInt__Times, LongInt__Mod, etc. > > > Maybe they should be in C:\dev2\cm3.2\m3-libs\m3core\src\types? > > > Similarly open question would be the set functions. > RTHooks.SetLT? > RTSet.LT? > C:\dev2\cm3.2\m3-libs\m3core\src\types\set? > C:\dev2\cm3.2\m3-libs\m3core\src\types\bigset? > (as opposed to word-sized set, besides that eq/ne are inlined by the frontend for "medium" sizes) > > > Is it clear what my questions are? > > > Let me try again. > Consider Long.Rotate. Assume it takes a bunch of code. > m3back could notice it is compiling Long__Rotate, and generate the code inline. > Otherwise it can "know" such a function exists and generate a call to it. > This is a reasonable seeming model that I just came up with. > > > That does double the paths in m3back, granted. > > For several such m3cg operations, there is already a natural function to treat this way -- stuff in interface Long. > > > However, for a few operations, there is not already a place -- signed operations and operations on large sets. > > > As well, there is a presumed question/answer/non-answer: everything works today. > But should it be changed? > Is it better to reduce/eliminate hand.c and instead teach the Modula-3 code in m3back how to generate assembly? > "Better" how? > More efficient? Maybe. The factor here is that hand.c may or may not be optimized, but m3back operates in a mode where it always optimizes to some extent. It generally generates code that is significantly better than unoptimized C, and significantly worse than optimized C, depending -- it does pretty well, but it never inlines, but it only does a certain small class of smart things. It keeps values in registers somewhat, it doesn't eliminate dead stores, doesn't unroll loops. > > > Another factor is the principle of having C code or not. > We vehemently agree that C code is preferred in m3core/unix. > But otherwise many people here kind of dislike or discourage C on principle/philosophical grounds. > I'm not sure I agree, however by agreeing, I introduce an interesting technical challenge. :) > > > Again, must we really call it interface long? > Does that really "sound like": > INTEGER is to Word as LONGINT is to x? > > LongWord maybe? > > > - Jay > > > > From: jay.krell at cornell.edu > > To: hosking at cs.purdue.edu > > CC: m3commit at elegosoft.com > > Subject: RE: [M3commit] CVS Update: cm3 > > Date: Thu, 11 Mar 2010 04:10:40 +0000 > > > > > > Long.LeftShift is not a great example. Sorry. > > Long.Shift, Long.Rotate, Long.Times, LongDivide. > > Those are or "could be" large. Currently the later > > three all call functions and are therefore short. > > > > > > Integer.Multiple, Divide, Remainder, similarly uncertain. > > They call functions, the functions aren't short. > > > > > > I was also thinking, some of these functions, esp. > > 64bit shift/rotate could be shorter as loops. > > But I think not worthwhile. > > > > > > To be clear, even though "parse.c" "never" generates > > function calls (except for set operations), the gcc > > backend itself often does. The functions are in libgcc. > > > > > > You know..I find it hard to decide about compiler generated > > function calls. Definitely for size optimization they > > can be a win. But I just somehow vaguely don't like them. > > In many cases, the decision is not difficult. set_singleton > > and set_member are good examples. They could be inlined > > and be very small. But Shift, Rotate, Multiply, Divide, > > it gets less clear. > > > > > > But no matter how large an inline form, I think > > when interface word/long maps directly to these operations, > > they should be inlined. > > > > > > And then if there are places that don't inline, they should > > call the "instantiations" in interface word/long. > > > > > > And then, furthermore, we should consider an interface > > for "Integer.Divide/Multiple/Remainder"? > > A place to park the possibly large code? > > > > > > - Jay > > > > > > ---------------------------------------- > > > From: hosking at cs.purdue.edu > > > Date: Wed, 10 Mar 2010 21:01:54 -0500 > > > To: jay.krell at cornell.edu > > > CC: m3commit at elegosoft.com > > > Subject: Re: [M3commit] CVS Update: cm3 > > > > > > I am confused. The intention is that Long.LeftShift is always inlined wherever it is called. Is the code for it really that hairy? > > > > > > On 10 Mar 2010, at 20:30, Jay K wrote: > > > > > >> > > >> If there exists an inline expansion, and it is large, but there is a function whose job it is to hold the expansion, whether it is inline or not, and the backend was informed of this function's name, and whether or not it was currently producing it, the backend could generate the inline expansion for Long__LeftShift, but otherwise generate calls to it. > > >> > > >> > > >> That is, e.g. m3cg.left_shift(type := int64|word64) could chose to either call Long__LeftShift or generate the body of Long__LeftShift. Not based on some "is inlining profitable heuristic", but specifically if told it is generating the function Long__LeftShift or not. > > >> > > >> > > >> That is, there's no point in having a C function m3_left_shift64 or somesuch, and having Long__LeftShift call it. Instead a backend should generate the body of Long__LeftShift inline when told it is generating that function, vs. generating a call to that function when it is otherwise asked to do a 64bit leftshift if it decides that inlining it every time is too large. > > >> > > >> Granted at least two things: > > >> Given 64bit target, Long__LeftShift vs. Word__LeftShift is ambiguous. > > >> And I'm inlined to just always inline anyway. > > >> > > >> > > >> (I still don't like the term "Long" here. It doesn't convey unsigned.) > > >> > > >> > > >> - Jay > > >> > > >> > > >> > > >> ________________________________ > > >>> Subject: Re: [M3commit] CVS Update: cm3 > > >>> From: hosking at cs.purdue.edu > > >>> Date: Wed, 10 Mar 2010 19:57:17 -0500 > > >>> CC: m3commit at elegosoft.com > > >>> To: jay.krell at cornell.edu > > >>> > > >>> > > >>> > > >>> Yes, someone can pass the function as a parameter. > > >>> > > >>> I don't understand the rest of what you are saying. > > >>> > > >>> > > >>> Antony Hosking | Associate Professor | Computer Science | Purdue University > > >>> 305 N. University Street | West Lafayette | IN 47907 | USA > > >>> Office +1 765 494 6001 | Mobile +1 765 427 5484 > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> On 10 Mar 2010, at 17:27, Jay K wrote: > > >>> > > >>> > > >>> Hey, should we maybe have a flag that indicates we are producing a function that is meant to provide and only provide exactly the functionality of the "one" m3cg call? > > >>> You know, that'd be a great hint to: > > >>> > > >>> - definitely definitely definitely definitely inline > > >>> - if we know this thing existed, we could use it instead of a .c file. > > >>> - along with the function's name for the invocations that aren't the function? > > >>> > > >>> > > >>> Why provide the function anyway? In case someone takes the address? > > >>> > > >>> > > >>> > > >>> - Jay > > >>> > > >>> > > >>> ________________________________ > > >>> From: jay.krell at cornell.edu > > >>> To: hosking at cs.purdue.edu > > >>> Date: Mon, 8 Mar 2010 16:04:45 +0000 > > >>> CC: m3commit at elegosoft.com > > >>> Subject: Re: [M3commit] CVS Update: cm3 > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> Fair enough. > > >>> > > >>> > > >>> > > >>> - Jay > > >>> > > >>> > > >>> > > >>> ________________________________ > > >>> > > >>> From: hosking at cs.purdue.edu > > >>> Date: Mon, 8 Mar 2010 10:29:00 -0500 > > >>> To: jay.krell at cornell.edu > > >>> CC: m3commit at elegosoft.com > > >>> Subject: Re: [M3commit] CVS Update: cm3 > > >>> > > >>> > > >>> > > >>> > > >>> You can't call the support function Long__Extract because we already have a Long__Extract in m3core/src/word. It is the library representative of the front-end compiler-generated (inlined) Long.Extract. > > >>> > > >>> > > >>> > > >>> On 8 Mar 2010, at 07:30, Jay K wrote: > > >>> > > >>> > > >>> > > >>> diff attached > > >>> > > >>> > > >>> If folks really want to use tables or function calls here, let me know. > > >>> > > >>> > > >>> The data was historically problematic, though we worked out the problems. > > >>> - It was initialized at runtime which has a race condition, fixed years ago. > > >>> - It can't be "exported" on NT386 (which is the only platform that uses it) and must be statically linked. > > >>> > > >>> The data is still used by Word.Insert and Long.Insert is still a function, but those > > >>> are my next targets. > > >>> > > >>> > > >>> This is a case where the user does write a function call. Word.Extract or Long.Extract. > > >>> (So the function should have been called Long__Extract.) > > >>> > > >>> > > >>> - Jay > > >>> > > >>> Date: Mon, 8 Mar 2010 13:26:36 +0000 > > >>> To: m3commit at elegosoft.com > > >>> From: jkrell at elego.de > > >>> Subject: [M3commit] CVS Update: cm3 > > >>> > > >>> CVSROOT: /usr/cvs > > >>> Changes by: jkrell at birch. 10/03/08 13:26:36 > > >>> > > >>> Modified files: > > >>> cm3/m3-sys/m3back/src/: M3x86.m3 Stackx86.i3 Stackx86.m3 > > >>> > > >>> Log message: > > >>> "rewrite" extract to not use tables and always be inlined for 64bit > > >>> > > >>> equivalent C code: > > >>> > > >>> UT __stdcall extract(UT x, uint32 offset, uint32 count) > > >>> { > > >>> x>>= offset; > > >>> x &= ~((~(UT)0) << count); > > >>> return x; > > >>> } > > >>> > > >>> extract32 > > >>> 00000729: 8B4DEC MOV ECX tv.126[_m] > > >>> 0000072C: 8B5DF4 MOV EBX tv.124[_a32] > > >>> 0000072F: D3EB SHR EBX > > >>> 00000731: 8BD1 MOV EDX ECX This is pointless and I'll try to remove. > > >>> 00000733: 8B4DE4 MOV ECX tv.128[_n] > > >>> 00000736: BEFFFFFFFF MOV ESI $4294967295 I'll look for smaller form of this. or esi -1? > > >>> 0000073B: D3E6 SHL ESI > > >>> 0000073D: F7D6 NOT ESI > > >>> 0000073F: 23DE AND EBX ESI > > >>> > > >>> extract64 > > >>> 000008E4: 8B4DD8 MOV ECX tv.134[_m] > > >>> 000008E7: 8B5DE8 MOV EBX tv.131[_a64] > > >>> 000008EA: 8B55EC MOV EDX tv.131[_a64]+4 > > >>> 000008ED: 0FADD3 SHRD EBX EDX ECX > > >>> 000008F0: D3EA SHR EDX > > >>> 000008F2: F6C120 TEST ECX $32 > > >>> 000008F5: 7400 JE rel8 L.107 > > >>> 000008F7: 8BDA MOV EBX EDX > > >>> 000008F9: 33D2 XOR EDX EDX > > >>> set_label L.107 > > >>> 000008FB: 8BF1 MOV ESI ECX This is pointless and I'll try to remove. > > >>> 000008FD: 8B4DD0 MOV ECX tv.136[_n] > > >>> 00000900: BFFFFFFFFF MOV EDI $4294967295 I'll look for smaller form of this. or edi -1? > > >>> 00000905: B8FFFFFFFF MOV EAX $4294967295 I'll look for smaller form of this. (heck, at least mov edi, eax) > > >>> 0000090A: 0FA5F8 SHLD EAX EDI ECX > > >>> 0000090D: D3E7 SHL EDI > > >>> 0000090F: F6C120 TEST ECX $32 > > >>> 00000912: 7400 JE rel8 L.108 > > >>> 00000914: 8BC7 MOV EAX EDI > > >>> 00000916: 33FF XOR EDI EDI > > >>> set_label L.108 > > >>> 00000918: F7D7 NOT EDI > > >>> 0000091A: F7D0 NOT EAX > > >>> 0000091C: 23DF AND EBX EDI > > >>> 0000091E: 23D0 AND EDX EAX > > >>> > > >>> having n or m and n (or just m? I think so.) be constant leads to better code > > >>> > > >>> some other small cleanup, like avoiding calling find twice, > > >>> I don't see why it was that way > > >>> > > >>> <1.txt> > > >>> > > >>> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Thu Mar 11 23:03:30 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 17:03:30 -0500 Subject: [M3devel] Target.EOL In-Reply-To: References: Message-ID: I don't think I've used Target.EOL, but I've definitely used some EOL definition somewhere, maybe it was Wr.EOL, I'd have to check to be sure. I'm not sure the implications of removing Target.EOL, but whatever is done, we need to maintain a way to distinguish at runtime the proper line ending format for the host environment. I've written a lot of code in the past where this is important. Some of my code interfaces with other systems whose line ending formats have to be compared and/or reconciled with the host when doing serial data transfer (yes there are still systems that rely on good 'ol serial I/O, particularly legacy DoD systems). Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 10:41 AM To: m3devel Subject: [M3devel] Target.EOL Importance: Low I strongly suspect that Target.EOL can go away and just use Wr.EOL instead. Or even just \n. Cross builds are fairly rare, esp. cross between NT and Posix, and very many tools treat \n, \r\n, and perhaps \r the same, so even crossing NT <=> Posix doesn't likely matter. ie. it works, assuming you have a cross m3cg (ie: mingwin/cygwin hosted for NT hosts) gcc, Visual C++ compiler, m3front, all treat \r and \r\n the same. (witness all our *.h, *.c, *.i3, *.m3 files use \r\n) I think we have some tools that don't understand \r\n, but in my opinion that is a bug. All three formats should be treated the same. I know notepad doesn't display \n well, and Sun cc might not like \r\n. I know some compiler I used recently was picky, but "many" compilers are not. cmd might be ok with \n. Python calls it "universal newlines", treating all three formats the same. C:\dev2\cm3.2\m3-sys\cm3\src\Utils.m3(190):PROCEDURE CopyText (old, new: TEXT) = C:\dev2\cm3.2\m3-sys\m3middle\src\M3File.m3(61):PROCEDURE CopyText (src, dest: TEXT; eol: TEXT) RAISES {OSError.E} = probably leave alone, except that they don't treat \r correctly if you consider the historical Apple behavior. There are three formats, not two. Anyway, it's not a big deal. I think my problem here is solved by the change to m3x86.m3 I already made. I'll put Target.EOL back on my machine. - Jay ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 12 01:25:15 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 00:25:15 +0000 Subject: [M3devel] Target.EOL In-Reply-To: References: , Message-ID: Serial I/O is used plenty, for debugging. Though the protocols are generally "binary" not "text". Even any world with multiple systems messes with historical notions that one system is one way, other systems are another, and that they rarely see each other or exchange files. File exchange is very common these days, so it behooves every piece of new code to understand either format and possibly detect what the old code on the other side wants and be willing to write any of the three formats. "Constants" end up not useful. This isn't the EOL you care about. Target.EOL is written to .m3ship, foo.molog. It is used in m3front, m3middle, m3back, cm3 packages. It is exposed from m3middle/Target.i3. It is used by the old bootstrapping code, which is strewn around cm3, which I do use some of as part of my automation, uses it. (I don't use e.g. the makefile fragments that it outputs, but I use the behavior that stops after producing assembly, doesn't run the assembler or linker or C compiler) For example if you cross from NT386 to Posix, any \r\n in .man, .c, .h files will be changed to \n, and copied to the target directory. The thing is though, we don't have any \r\n anyway so that is a no-op. If you cross from Posix to NT386, \n will changed to \r\n. But it doesn't matter. Native builds just leave the \n alone -- which is what we always have -- and they work fine. If you cross from a hypothetical old Macintosh with its \r format, the newlines get completely destroyed because the code is wrong. Oops. Target.EOL can be changed to Wr.EOL or a hardcoded \n and almost nothing would change. The difference is that if we hardcoded \n, native NT386 users who happened to look at .m3ship with notepad, wouldn't have a good experience. Almost nobody ever looks at those files. Use Wr.EOL and then the only downside would be .m3ship files when crossing to NT386. And even still, .m3ship files don't participate in cross builds as I do them. My cross build only produces cm3, and then I build the rest native. Though there is a place for this perhaps. A cross build that builds everything, leaving final assembly, C compilation, linking and shipping to the target. We should consider that as a future distribution format. Similar to today's packages, but cross. But still, Wr.EOL will be fine. I'm very tempted to say the same thing about / and \, except that I recently experienced the pain of using an older NT386 cm3 that doesn't accept \. It behooves me to use \ "where appropriate" so I can be compatible with that. I need to fix scripts/python and/or cminstall/src/config-no-install a little. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Thu, 11 Mar 2010 17:03:30 -0500 Subject: RE: [M3devel] Target.EOL I don?t think I?ve used Target.EOL, but I?ve definitely used some EOL definition somewhere, maybe it was Wr.EOL, I?d have to check to be sure. I?m not sure the implications of removing Target.EOL, but whatever is done, we need to maintain a way to distinguish at runtime the proper line ending format for the host environment. I?ve written a lot of code in the past where this is important. Some of my code interfaces with other systems whose line ending formats have to be compared and/or reconciled with the host when doing serial data transfer (yes there are still systems that rely on good ?ol serial I/O, particularly legacy DoD systems). Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 10:41 AM To: m3devel Subject: [M3devel] Target.EOL Importance: Low I strongly suspect that Target.EOL can go away and just use Wr.EOL instead. Or even just \n. Cross builds are fairly rare, esp. cross between NT and Posix, and very many tools treat \n, \r\n, and perhaps \r the same, so even crossing NT <=> Posix doesn't likely matter. ie. it works, assuming you have a cross m3cg (ie: mingwin/cygwin hosted for NT hosts) gcc, Visual C++ compiler, m3front, all treat \r and \r\n the same. (witness all our *.h, *.c, *.i3, *.m3 files use \r\n) I think we have some tools that don't understand \r\n, but in my opinion that is a bug. All three formats should be treated the same. I know notepad doesn't display \n well, and Sun cc might not like \r\n. I know some compiler I used recently was picky, but "many" compilers are not. cmd might be ok with \n. Python calls it "universal newlines", treating all three formats the same. C:\dev2\cm3.2\m3-sys\cm3\src\Utils.m3(190):PROCEDURE CopyText (old, new: TEXT) = C:\dev2\cm3.2\m3-sys\m3middle\src\M3File.m3(61):PROCEDURE CopyText (src, dest: TEXT; eol: TEXT) RAISES {OSError.E} = probably leave alone, except that they don't treat \r correctly if you consider the historical Apple behavior. There are three formats, not two. Anyway, it's not a big deal. I think my problem here is solved by the change to m3x86.m3 I already made. I'll put Target.EOL back on my machine. - Jay CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Highjinks at gmx.com Fri Mar 12 01:35:25 2010 From: Highjinks at gmx.com (Chris) Date: Fri, 12 Mar 2010 01:35:25 +0100 (CET) Subject: [M3devel] Constant to Constant? Message-ID: <20100311203930.9b7cb406.Highjinks@gmx.com> Things are progressing here, slowly but steadily. I'm curious... When writing an interface for something like this ... /* C Type */ const C_Foo_t *Bar = get_foo_data(misc); Is it better to do it this way... <* EXTERNAL get_foo_data:C *> TYPE Foo_Data = PROCEDURE(get_foo_data(somemisc : misctype) : C_Foo_t; Or should I just do... UNSAFE INTERFACE Foo; (* Translating the C typedef struct over to Modula3 code. *) TYPE C_Foo_T = UNTRACED REF RECORD .... END; <* EXTERNAL get_foo_data:C *> PROCEDURE get_foo_data(somemisc : misctype) : C_foo_t; END Foo. Main.m3 IMPORT Foo; Foo_Data := Foo.get_foo_data(Misc); END Main. C_Foo_t is known, and has a Modula 3 representation in the Interface. But as the C Code shows, it has to be a constant value. What's the best way to do this without hosing the Interface for everyone else that might want to use it? Variable expressions are no problem, it's making it a constant that's giving me trouble. It's constant because the data structure is managed by the supporting library, not by the Modula3 Code. Tips...pointers? -- Chris From jay.krell at cornell.edu Fri Mar 12 02:06:57 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 01:06:57 +0000 Subject: [M3devel] Constant to Constant? In-Reply-To: <20100311203930.9b7cb406.Highjinks@gmx.com> References: <20100311203930.9b7cb406.Highjinks@gmx.com> Message-ID: Chris, I'm not sure I understand. Is the result "in misc", or is it constant relative to the program's lifetime? Or is it initialized on-demand and then constant from then on? There are a few instances where "constants" are initialized in Modula-3 module initializers. They are "VAR" from a technical point of view, in the .i3 file, but then VAR is immediately preceded by a comment that they are CONST. I do something similar in m3core/unix, except the data is actually const/static initialized in C code. I do that to avoid duplicating header content. Header content can be duplicated, if it is fairly portable and stable, only needs to be duplicated once, easy to get correct, and stay correct. The particular problem in m3core/unix is portability -- the header content would have to be duplicated differently for every target. Incorrectly duplicated C headers silently violate safety (see below). You should try to provide a SAFE interface, so that client code can be SAFE. And then, you are responsible for upholding what that means. Your module implementation might be UNSAFE. But clients must be protected from a "certain class of bug". For example, a claimed-to-be SAFE interface cannot expose stdlib/free() because a client could double free. This is an easy thing to mess up and it is very unfortunate when it is. It is roughly equivalent to bugs in the compiler or garbage collector. - Jay > From: Highjinks at gmx.com > To: m3devel at elegosoft.com > Date: Fri, 12 Mar 2010 01:35:25 +0100 > Subject: [M3devel] Constant to Constant? > > > Things are progressing here, slowly but steadily. > > I'm curious... > > When writing an interface for something like this ... > > /* C Type */ > const C_Foo_t *Bar = get_foo_data(misc); > > Is it better to do it this way... > > <* EXTERNAL get_foo_data:C *> > TYPE Foo_Data = PROCEDURE(get_foo_data(somemisc : misctype) : C_Foo_t; > > Or should I just do... > UNSAFE INTERFACE Foo; > > (* Translating the C typedef struct over to Modula3 code. *) > TYPE C_Foo_T = UNTRACED REF RECORD .... END; > > <* EXTERNAL get_foo_data:C *> > PROCEDURE get_foo_data(somemisc : misctype) : C_foo_t; > END Foo. > > Main.m3 > IMPORT Foo; > Foo_Data := Foo.get_foo_data(Misc); > END Main. > > C_Foo_t is known, and has a Modula 3 representation in the Interface. But as the C Code shows, it has to be a constant value. > > What's the best way to do this without hosing the Interface for everyone else that might want to use it? Variable expressions are no problem, it's making it a constant that's giving me trouble. It's constant because the data structure is managed by the supporting library, not by the Modula3 Code. > > Tips...pointers? > > -- > Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Mar 12 02:14:12 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 20:14:12 -0500 Subject: [M3devel] Ticket #1082 Message-ID: I have updated the comments on Ticket #1082 There are two postings, as shown below. Note that it looks like there is a problem with "START /WAIT ..." command differences between Windows versions, so we may need to open up a new Trac ticket for this issue (see comment #2 below). FIRST COMMENT: --------------------- The "startReactor.CMD" file is obsolete and no longer in the repository. The new files are in "scripts\install\windows" and should be copied to the "cm3\bin" folder on a windows installation. "cm3StartIDE.CMD" replaces "startReactor.CMD". This file makes use of "cm3CommandShell.CMD", a script that ensures the environment is set up properly, including Visual C++. It appears that the problem reporter has his installation rooted at E:\cm3 rather than C:\cm3. So, he will need to set an environment variable "set CM3_ROOT=E:\cm3" to let these scripts know where to find the cm3 installation. Alternately, cm3CommandShell can be invoked with the arguments "Root E:\cm3" to tell it where to find the installation. Today, I checked in an update to cm3CommandShell.CMD that allows it to search the current PATH environment variable in an attempt to find the root of the cm3 installation, so now if "E:\cm3\bin" is in your path, it will find it without having to set the CM3_ROOT or invoking with "Root E:\cm3". Hope this helps! Note also that strictly speaking, these CMD files are not necessary; rather they are conveniences I've found useful under Windows. I also can supply some .REG files that will allow you to integrate these into the Explorer shell so that you can start a cm3 command shell in any folder and you can start CM3IDE. SECOND COMMENT: --------------------- Ok as for CM3IDE timing out, I've been able to reproduce this problem on Vista. There seems to be some difference I don't understand yet between the way the various Windows versions deal with the "START /WAIT ..." command. Here is a "quick fix" until we come up with a better solution. Note that the downside here is that when you terminate your browser, CM3IDE will stay running, instead of terminating. You can restart your browser and point it back to the http://localhost:3800/ URL and verify that it reconnects to CM3IDE. So, to terminate CM3IDE, you will have to use CTRL-C from its console output window. Now for the quick fix: 1. Go to your CM3_IDE_HOME folder (e.g., C:\Users\RColeburn\Documents\CM3_IDE_Home). 2. Open the most recent CM3_IDE.cfg file in notepad (e.g., notepad CM3_IDE.cfg1). 3. Edit the line toward the end of the file that begins with "proc start_browser". On this line change TRUE to FALSE. Note that case is significant. Now try cm3IDE.exe, or cm3_StartIDE.CMD. It should work for you now as described above. Regards, Randy Coleburn ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Mar 12 02:38:34 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 20:38:34 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait Message-ID: Ok, I've done a bit more testing. I don't think the problem has to do with "START /WAIT ..." but rather it seems to be a difference in Windows Internet Explorer. When your browser is set to FIREFOX instead of IE, the "START /WAIT firefox.exe" command works fine, so you can keep the "return TRUE" in the CM3_IDE.cfg file. But, when using IE8, the "START /WAIT iexplorer.exe" command does not wait for IE8 to terminate and returns immediately, thus the "return TRUE" in CM3_IDE.cfg must be changed to "return FALSE" to prevent immediate shutdown of CM3IDE. If I go to a Windows 2000 box running IE6, the "START /WAIT" again seems to wait for IE to terminate. I haven't tested yet with IE7 to see if it behaves correctly or not. So for now, folks may want to use FireFox instead of IE with CM3IDE. Regards, Randy ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 12 02:43:03 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 01:43:03 +0000 Subject: [M3devel] more info on ticket 1082 wrt start /wait In-Reply-To: References: Message-ID: What if you just simply: "start http://localhost:whateverport" Personally I find cm3ide/reactor to be ..not great. The behavior you are noticing is that the web browser might decide there is another adequate instance of it running and ask it to display something and exit. You will likely find different behavior depending on if an instance of the browser is already running. There might also be a command line option to control this. But I think your best best is just not to use /wait. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Thu, 11 Mar 2010 20:38:34 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait Ok, I?ve done a bit more testing. I don?t think the problem has to do with ?START /WAIT ?? but rather it seems to be a difference in Windows Internet Explorer. When your browser is set to FIREFOX instead of IE, the ?START /WAIT firefox.exe? command works fine, so you can keep the ?return TRUE? in the CM3_IDE.cfg file. But, when using IE8, the ?START /WAIT iexplorer.exe? command does not wait for IE8 to terminate and returns immediately, thus the ?return TRUE? in CM3_IDE.cfg must be changed to ?return FALSE? to prevent immediate shutdown of CM3IDE. If I go to a Windows 2000 box running IE6, the ?START /WAIT? again seems to wait for IE to terminate. I haven?t tested yet with IE7 to see if it behaves correctly or not. So for now, folks may want to use FireFox instead of IE with CM3IDE. Regards, Randy CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 12 02:36:53 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 01:36:53 +0000 Subject: [M3devel] Ticket #1082 In-Reply-To: References: Message-ID: > There seems to be some difference I don't understand yet between > the way the various Windows versions deal with the "START /WAIT ..." command. Please elaborate. What varying behaviors are you seeing on different versions? I have used start /wait a fair number of times and never noticed any differences among Windows versions, at least on NT versions since circa NT 4. Definitely maybe some versions don't have any sort of start nor /wait, and start on Win9x is probably different than NT. start is a builtin command to NT cmd but an .exe on Win9x, as I vaguely recall. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Thu, 11 Mar 2010 20:14:12 -0500 Subject: [M3devel] Ticket #1082 I have updated the comments on Ticket #1082 There are two postings, as shown below. Note that it looks like there is a problem with ?START /WAIT ?? command differences between Windows versions, so we may need to open up a new Trac ticket for this issue (see comment #2 below). FIRST COMMENT: --------------------- The "startReactor.CMD" file is obsolete and no longer in the repository. The new files are in "scripts\install\windows" and should be copied to the "cm3\bin" folder on a windows installation. "cm3StartIDE.CMD" replaces "startReactor.CMD". This file makes use of "cm3CommandShell.CMD", a script that ensures the environment is set up properly, including Visual C++. It appears that the problem reporter has his installation rooted at E:\cm3 rather than C:\cm3. So, he will need to set an environment variable "set CM3_ROOT=E:\cm3" to let these scripts know where to find the cm3 installation. Alternately, cm3CommandShell can be invoked with the arguments "Root E:\cm3" to tell it where to find the installation. Today, I checked in an update to cm3CommandShell.CMD that allows it to search the current PATH environment variable in an attempt to find the root of the cm3 installation, so now if "E:\cm3\bin" is in your path, it will find it without having to set the CM3_ROOT or invoking with "Root E:\cm3". Hope this helps! Note also that strictly speaking, these CMD files are not necessary; rather they are conveniences I?ve found useful under Windows. I also can supply some .REG files that will allow you to integrate these into the Explorer shell so that you can start a cm3 command shell in any folder and you can start CM3IDE. SECOND COMMENT: --------------------- Ok as for CM3IDE timing out, I've been able to reproduce this problem on Vista. There seems to be some difference I don't understand yet between the way the various Windows versions deal with the "START /WAIT ..." command. Here is a "quick fix" until we come up with a better solution. Note that the downside here is that when you terminate your browser, CM3IDE will stay running, instead of terminating. You can restart your browser and point it back to the http://localhost:3800/ URL and verify that it reconnects to CM3IDE. So, to terminate CM3IDE, you will have to use CTRL-C from its console output window. Now for the quick fix: 1. Go to your CM3_IDE_HOME folder (e.g., C:\Users\RColeburn\Documents\CM3_IDE_Home). 2. Open the most recent CM3_IDE.cfg file in notepad (e.g., notepad CM3_IDE.cfg1). 3. Edit the line toward the end of the file that begins with "proc start_browser". On this line change TRUE to FALSE. Note that case is significant. Now try cm3IDE.exe, or cm3_StartIDE.CMD. It should work for you now as described above. Regards, Randy Coleburn CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Mar 12 02:46:13 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 20:46:13 -0500 Subject: [M3devel] Constant to Constant? In-Reply-To: <20100311203930.9b7cb406.Highjinks@gmx.com> References: <20100311203930.9b7cb406.Highjinks@gmx.com> Message-ID: Chris: You may also want to look at pkg\m3core\src\win32\WinUser.i3 Regards, Randy -----Original Message----- From: Chris [mailto:Highjinks at gmx.com] Sent: Thursday, March 11, 2010 7:35 PM To: m3devel at elegosoft.com Subject: [M3devel] Constant to Constant? Things are progressing here, slowly but steadily. I'm curious... When writing an interface for something like this ... /* C Type */ const C_Foo_t *Bar = get_foo_data(misc); Is it better to do it this way... <* EXTERNAL get_foo_data:C *> TYPE Foo_Data = PROCEDURE(get_foo_data(somemisc : misctype) : C_Foo_t; Or should I just do... UNSAFE INTERFACE Foo; (* Translating the C typedef struct over to Modula3 code. *) TYPE C_Foo_T = UNTRACED REF RECORD .... END; <* EXTERNAL get_foo_data:C *> PROCEDURE get_foo_data(somemisc : misctype) : C_foo_t; END Foo. Main.m3 IMPORT Foo; Foo_Data := Foo.get_foo_data(Misc); END Main. C_Foo_t is known, and has a Modula 3 representation in the Interface. But as the C Code shows, it has to be a constant value. What's the best way to do this without hosing the Interface for everyone else that might want to use it? Variable expressions are no problem, it's making it a constant that's giving me trouble. It's constant because the data structure is managed by the supporting library, not by the Modula3 Code. Tips...pointers? -- Chris CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. From rcolebur at SCIRES.COM Fri Mar 12 03:06:36 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 21:06:36 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait In-Reply-To: References: Message-ID: Jay et al: Well I've done more testing. Jay you are right about the multiple instances. For both IE8 and FireFox on Vista, if you already have an instance running and use "START /WAIT" to get another instance, the second instance starts up (a new window) and the "START /WAIT" immediately returns rather than waiting. Conversely, if this is the first instance of IE8 or FireFox, then "START /WAIT" does indeed wait for the browser to close before it returns. I'll have to get to an XP machine to confirm if this same behavior occurs there as well. If you go back to IE6, it didn't have Tabbed windows, so maybe that is why "START /WAIT" always waited. I don't have an immediate fix for this behavior since the original design used the "START /WAIT" functionality on Windows. On POSIX, it doesn't. The quick fix I mentioned earlier is probably the best thing to do now. I could perhaps modify the sources to change the default CFG to "return FALSE" on Windows instead of "return TRUE", thereby implementing the quick fix for everyone. I'll check into this shortly. As for your comments about cm3ide not being "great" that is your opinion; however, back in the day when reactor (aka cm3ide) first came out, this was indeed leading edge stuff. The idea that you could tweak your IDE by making simple HTML file changes I think was pretty novel at the time, and the thinking from the CMass Inc folks was that as the browser improved, your IDE would improve automatically (e.g. multiple tabs was a subsequent browser improvement). Even though you may not consider CM3IDE to be a great IDE, I do find that it is extremely helpful during development when browsing source code and looking up interface specs. With just a hyperlink, you can quickly find what you are looking for. It also has a lot of built-in links to reference material and coding examples. Note that cm3ide hasn't really changed much at all since reactor first debuted. Now that we finally have the open-source release, we can modify it to make it better. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 8:43 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait What if you just simply: "start http://localhost:whateverport" Personally I find cm3ide/reactor to be ..not great. The behavior you are noticing is that the web browser might decide there is another adequate instance of it running and ask it to display something and exit. You will likely find different behavior depending on if an instance of the browser is already running. There might also be a command line option to control this. But I think your best best is just not to use /wait. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Thu, 11 Mar 2010 20:38:34 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait Ok, I've done a bit more testing. I don't think the problem has to do with "START /WAIT ..." but rather it seems to be a difference in Windows Internet Explorer. When your browser is set to FIREFOX instead of IE, the "START /WAIT firefox.exe" command works fine, so you can keep the "return TRUE" in the CM3_IDE.cfg file. But, when using IE8, the "START /WAIT iexplorer.exe" command does not wait for IE8 to terminate and returns immediately, thus the "return TRUE" in CM3_IDE.cfg must be changed to "return FALSE" to prevent immediate shutdown of CM3IDE. If I go to a Windows 2000 box running IE6, the "START /WAIT" again seems to wait for IE to terminate. I haven't tested yet with IE7 to see if it behaves correctly or not. So for now, folks may want to use FireFox instead of IE with CM3IDE. Regards, Randy ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 12 03:47:53 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 11 Mar 2010 21:47:53 -0500 Subject: [M3devel] Target.EOL In-Reply-To: References: Message-ID: <5F2763E1-801E-4EB2-BFAC-884EB53DC3C6@cs.purdue.edu> On 11 Mar 2010, at 17:03, Coleburn, Randy wrote: > I don?t think I?ve used Target.EOL, but I?ve definitely used some EOL definition somewhere, maybe it was Wr.EOL, I?d have to check to be sure. This is part of the compiler subsystem and its tool interfaces. I don't think it will affect application code. If we are to default I suggest we go with the more compact \n termination instead of verbose \r\n. > > I?m not sure the implications of removing Target.EOL, but whatever is done, we need to maintain a way to distinguish at runtime the proper line ending format for the host environment. > > I?ve written a lot of code in the past where this is important. Some of my code interfaces with other systems whose line ending formats have to be compared and/or reconciled with the host when doing serial data transfer (yes there are still systems that rely on good ?ol serial I/O, particularly legacy DoD systems). > > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Thursday, March 11, 2010 10:41 AM > To: m3devel > Subject: [M3devel] Target.EOL > Importance: Low > > I strongly suspect that Target.EOL can go away and just use Wr.EOL instead. > Or even just \n. > > > Cross builds are fairly rare, esp. cross between NT and Posix, and very > many tools treat \n, \r\n, and perhaps \r the same, so even > crossing NT <=> Posix doesn't likely matter. > ie. it works, assuming you have a cross m3cg (ie: mingwin/cygwin hosted for NT hosts) > > > gcc, Visual C++ compiler, m3front, all treat \r and \r\n the same. > (witness all our *.h, *.c, *.i3, *.m3 files use \r\n) > I think we have some tools that don't understand \r\n, but in my opinion that is a bug. > All three formats should be treated the same. > > > I know notepad doesn't display \n well, and Sun cc might not like \r\n. > I know some compiler I used recently was picky, but "many" compilers are not. > cmd might be ok with \n. > Python calls it "universal newlines", treating all three formats the same. > > > C:\dev2\cm3.2\m3-sys\cm3\src\Utils.m3(190):PROCEDURE CopyText (old, new: TEXT) = > C:\dev2\cm3.2\m3-sys\m3middle\src\M3File.m3(61):PROCEDURE CopyText (src, dest: TEXT; eol: TEXT) RAISES {OSError.E} = > > > probably leave alone, except that they don't treat \r correctly if you consider the historical Apple behavior. > There are three formats, not two. > > > Anyway, it's not a big deal. I think my problem here is solved by the change to m3x86.m3 I already made. > I'll put Target.EOL back on my machine. > > > - Jay > > CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. > > EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 12 03:49:39 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 02:49:39 +0000 Subject: [M3devel] more info on ticket 1082 wrt start /wait In-Reply-To: References: , , Message-ID: Start /wait is always doing the same thing, it is always waiting. But what it is launching is deciding to exit right away, or not. More later.. From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Thu, 11 Mar 2010 21:06:36 -0500 Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait Jay et al: Well I?ve done more testing. Jay you are right about the multiple instances. For both IE8 and FireFox on Vista, if you already have an instance running and use ?START /WAIT? to get another instance, the second instance starts up (a new window) and the ?START /WAIT? immediately returns rather than waiting. Conversely, if this is the first instance of IE8 or FireFox, then ?START /WAIT? does indeed wait for the browser to close before it returns. I?ll have to get to an XP machine to confirm if this same behavior occurs there as well. If you go back to IE6, it didn?t have Tabbed windows, so maybe that is why ?START /WAIT? always waited. I don?t have an immediate fix for this behavior since the original design used the ?START /WAIT? functionality on Windows. On POSIX, it doesn?t. The quick fix I mentioned earlier is probably the best thing to do now. I could perhaps modify the sources to change the default CFG to ?return FALSE? on Windows instead of ?return TRUE?, thereby implementing the quick fix for everyone. I?ll check into this shortly. As for your comments about cm3ide not being ?great? that is your opinion; however, back in the day when reactor (aka cm3ide) first came out, this was indeed leading edge stuff. The idea that you could tweak your IDE by making simple HTML file changes I think was pretty novel at the time, and the thinking from the CMass Inc folks was that as the browser improved, your IDE would improve automatically (e.g. multiple tabs was a subsequent browser improvement). Even though you may not consider CM3IDE to be a great IDE, I do find that it is extremely helpful during development when browsing source code and looking up interface specs. With just a hyperlink, you can quickly find what you are looking for. It also has a lot of built-in links to reference material and coding examples. Note that cm3ide hasn?t really changed much at all since reactor first debuted. Now that we finally have the open-source release, we can modify it to make it better. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 8:43 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait What if you just simply: "start http://localhost:whateverport" Personally I find cm3ide/reactor to be ..not great. The behavior you are noticing is that the web browser might decide there is another adequate instance of it running and ask it to display something and exit. You will likely find different behavior depending on if an instance of the browser is already running. There might also be a command line option to control this. But I think your best best is just not to use /wait. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Thu, 11 Mar 2010 20:38:34 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait Ok, I?ve done a bit more testing. I don?t think the problem has to do with ?START /WAIT ?? but rather it seems to be a difference in Windows Internet Explorer. When your browser is set to FIREFOX instead of IE, the ?START /WAIT firefox.exe? command works fine, so you can keep the ?return TRUE? in the CM3_IDE.cfg file. But, when using IE8, the ?START /WAIT iexplorer.exe? command does not wait for IE8 to terminate and returns immediately, thus the ?return TRUE? in CM3_IDE.cfg must be changed to ?return FALSE? to prevent immediate shutdown of CM3IDE. If I go to a Windows 2000 box running IE6, the ?START /WAIT? again seems to wait for IE to terminate. I haven?t tested yet with IE7 to see if it behaves correctly or not. So for now, folks may want to use FireFox instead of IE with CM3IDE. Regards, Randy CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Mar 12 04:37:48 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 11 Mar 2010 22:37:48 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait In-Reply-To: References: , , Message-ID: Jay: Yes, you said it better than I did. The problem is with the browser behavior, not with START /WAIT. I've implemented a quick-fix in the source code to change the default behavior on Windows to "return FALSE" on the "start_browser" function instead of "return TRUE". That way, regardless of whether or not the browser exits right away, CM3IDE will stay running until it is manually terminated via CTRL-C in its console output window. (Note: "return TRUE" means that CM3IDE will terminate when the browser terminates, whereas "return FALSE" means that CM3IDE will stay running even after the browser terminates. On a server system where you have multiple network clients connecting to the single CM3IDE instance running on the server, you would always want to use "return FALSE". Likewise, on a Unix system where you start CM3IDE as a background process when the system boots, you would want to use "return FALSE".) I am already thinking of alternate solutions and an easy way for CM3IDE to limit itself to a single instance. I'll work on polishing these up in the next few days, but the quick-fix should solve the immediate problem for now. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 9:50 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait Start /wait is always doing the same thing, it is always waiting. But what it is launching is deciding to exit right away, or not. More later.. ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Thu, 11 Mar 2010 21:06:36 -0500 Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait Jay et al: Well I've done more testing. Jay you are right about the multiple instances. For both IE8 and FireFox on Vista, if you already have an instance running and use "START /WAIT" to get another instance, the second instance starts up (a new window) and the "START /WAIT" immediately returns rather than waiting. Conversely, if this is the first instance of IE8 or FireFox, then "START /WAIT" does indeed wait for the browser to close before it returns. I'll have to get to an XP machine to confirm if this same behavior occurs there as well. If you go back to IE6, it didn't have Tabbed windows, so maybe that is why "START /WAIT" always waited. I don't have an immediate fix for this behavior since the original design used the "START /WAIT" functionality on Windows. On POSIX, it doesn't. The quick fix I mentioned earlier is probably the best thing to do now. I could perhaps modify the sources to change the default CFG to "return FALSE" on Windows instead of "return TRUE", thereby implementing the quick fix for everyone. I'll check into this shortly. As for your comments about cm3ide not being "great" that is your opinion; however, back in the day when reactor (aka cm3ide) first came out, this was indeed leading edge stuff. The idea that you could tweak your IDE by making simple HTML file changes I think was pretty novel at the time, and the thinking from the CMass Inc folks was that as the browser improved, your IDE would improve automatically (e.g. multiple tabs was a subsequent browser improvement). Even though you may not consider CM3IDE to be a great IDE, I do find that it is extremely helpful during development when browsing source code and looking up interface specs. With just a hyperlink, you can quickly find what you are looking for. It also has a lot of built-in links to reference material and coding examples. Note that cm3ide hasn't really changed much at all since reactor first debuted. Now that we finally have the open-source release, we can modify it to make it better. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Thursday, March 11, 2010 8:43 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] more info on ticket 1082 wrt start /wait What if you just simply: "start http://localhost:whateverport" Personally I find cm3ide/reactor to be ..not great. The behavior you are noticing is that the web browser might decide there is another adequate instance of it running and ask it to display something and exit. You will likely find different behavior depending on if an instance of the browser is already running. There might also be a command line option to control this. But I think your best best is just not to use /wait. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Thu, 11 Mar 2010 20:38:34 -0500 Subject: [M3devel] more info on ticket 1082 wrt start /wait Ok, I've done a bit more testing. I don't think the problem has to do with "START /WAIT ..." but rather it seems to be a difference in Windows Internet Explorer. When your browser is set to FIREFOX instead of IE, the "START /WAIT firefox.exe" command works fine, so you can keep the "return TRUE" in the CM3_IDE.cfg file. But, when using IE8, the "START /WAIT iexplorer.exe" command does not wait for IE8 to terminate and returns immediately, thus the "return TRUE" in CM3_IDE.cfg must be changed to "return FALSE" to prevent immediate shutdown of CM3IDE. If I go to a Windows 2000 box running IE6, the "START /WAIT" again seems to wait for IE to terminate. I haven't tested yet with IE7 to see if it behaves correctly or not. So for now, folks may want to use FireFox instead of IE with CM3IDE. Regards, Randy ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 12 07:45:07 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 12 Mar 2010 06:45:07 +0000 Subject: [M3devel] merits/demerits of cm3ide In-Reply-To: References: , , Message-ID: > cm3ide > more later Being at the mercy of the rapidly changing web browsers is a very sharp double edged sword. An IDE with no editor and no debugger..is not an IDE. Maybe it is a source browser. Maybe an IDE is not worth it. People are very slow to change editors. Debuggers are hard to write. Probably better to try integrating with an actual IDE like VisualStudio or Eclipse. Or to have a decent enough language, library, build system, that the language doesn't need such costly support. Maybe we do have such a think. Hyperlinking the source is nice, but all I need is "find in files". I can open the documentation from a file system browser, which open it in a web browser. I should just be able to use online documentation anyway, not local stuff. Having the real compiler output enough information such that writing other lanugage-aware tools, such as hyperlinked source generation, is also good. Too many systems have their IDE write another parser, and no two parsers agree. However it is a tough problem, because the compiler's job is different than a source browser. For example, a compiler must reject incorrect source, but a source browser should be lenient. Relying on the browser is a cheap way to get a somewhat decent somewhat portable very limited gui. That's about it. I don't think it is fixable. It's just pretty much all wierd and wrong. I was quite shocked the first time I used it, having heard it described as an IDE. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Mar 12 10:20:59 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Fri, 12 Mar 2010 04:20:59 -0500 Subject: [M3devel] merits/demerits of cm3ide In-Reply-To: References: , , Message-ID: Jay as far as editor goes, CM3IDE lets you use the editor of your choice. Once you define it, CM3IDE calls upon your defined editor when you want to edit a source file. Note also that when you compile and get errors, CM3IDE annotates the listing and gives hyperlinks to allow you to make edits. When you click on a line with a syntax error and choose to edit it, CM3IDE calls upon your editor and tells it to position the cursor at that line so you can make the fix easily. CM3IDE also shows for each interface all modules that import that interface. This is useful also when you are making an interface change that will break modules, you can quickly find all the ones affected and make the edits. Granted you can use command line tools for finding stuff in files, but I personally like the hyper linking. In my environment, I use CM3IDE, the CRiSP programmers editor, and TortoiseCVS. CRiSP integrates with CVS so you can check files in/out of the repository automatically, all controlled from CM3IDE. There is no point in arguing further about merits/demerits of CM3IDE. I didn't write it; I simply worked to make it open-source. If you don't like it, don't use it. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, March 12, 2010 1:45 AM To: Coleburn, Randy; m3devel Subject: merits/demerits of cm3ide > cm3ide > more later Being at the mercy of the rapidly changing web browsers is a very sharp double edged sword. An IDE with no editor and no debugger..is not an IDE. Maybe it is a source browser. Maybe an IDE is not worth it. People are very slow to change editors. Debuggers are hard to write. Probably better to try integrating with an actual IDE like VisualStudio or Eclipse. Or to have a decent enough language, library, build system, that the language doesn't need such costly support. Maybe we do have such a think. Hyperlinking the source is nice, but all I need is "find in files". I can open the documentation from a file system browser, which open it in a web browser. I should just be able to use online documentation anyway, not local stuff. Having the real compiler output enough information such that writing other lanugage-aware tools, such as hyperlinked source generation, is also good. Too many systems have their IDE write another parser, and no two parsers agree. However it is a tough problem, because the compiler's job is different than a source browser. For example, a compiler must reject incorrect source, but a source browser should be lenient. Relying on the browser is a cheap way to get a somewhat decent somewhat portable very limited gui. That's about it. I don't think it is fixable. It's just pretty much all wierd and wrong. I was quite shocked the first time I used it, having heard it described as an IDE. - Jay ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Fri Mar 12 15:17:34 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 12 Mar 2010 15:17:34 +0100 Subject: [M3devel] merits/demerits of cm3ide In-Reply-To: References: , , Message-ID: <20100312151734.r2i0ui5nkg0k0woc@mail.elegosoft.com> Well, some people have used Reactor/cm3ide and liked it. There have been several requests for it in recent years. It's certainly no match for a large IDE framework like Eclipse, but it's the only GUI for the compiler we have. As for it not being an IDE, I do not really agree; it should already integrate compiling, shipping, source viewing, searching, pretty printing. And it integrates on the level of language elements likes types, interfaces etc. We should also be able to integrate a debugger (at least m3gdb); this has been done to gdb with other GUIs, too. I'd also like to see an M3 plugin for Eclipse, but nobody seems to be working on that. I myself am still happy with my XEmacs editor whenever I actually get to browse or write code ;-) So just view it as an offer and keep not using it if you don't like it. There are so many things that need work in the cm3 system that work won't get short during the next years. Olaf Quoting Jay K : > > cm3ide > > more later > > Being at the mercy of the rapidly changing web browsers is a very > sharp double edged sword. > > An IDE with no editor and no debugger..is not an IDE. > Maybe it is a source browser. > > Maybe an IDE is not worth it. > > People are very slow to change editors. > > Debuggers are hard to write. > > Probably better to try integrating with an actual IDE like > VisualStudio or Eclipse. > > Or to have a decent enough language, library, build system, that > the language doesn't > need such costly support. Maybe we do have such a think. > > Hyperlinking the source is nice, but all I need is "find in files". > I can open the documentation from a file system browser, which > open it in a web browser. I should just be able to use online > documentation anyway, not local stuff. > > Having the real compiler output enough information > such that writing other lanugage-aware tools, such > as hyperlinked source generation, is also good. > Too many systems have their IDE write another parser, > and no two parsers agree. However it is a tough problem, > because the compiler's job is different than a source > browser. For example, a compiler must reject incorrect > source, but a source browser should be lenient. > > Relying on the browser is a cheap way to get a somewhat > decent somewhat portable very limited gui. > That's about it. > > I don't think it is fixable. > It's just pretty much all wierd and wrong. > > I was quite shocked the first time I used it, having > heard it described as an IDE. -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From rcolebur at SCIRES.COM Sat Mar 13 06:57:00 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sat, 13 Mar 2010 00:57:00 -0500 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20100313011125.34CF12474001@birch.elegosoft.com> Message-ID: Jay: This is example code. The program terminates after the message box is displayed. There is no further processing. I did not write this example; I merely fixed it to compile again after someone changed the names of the procedures in the M3toC interface. Nevertheless, it is probably a bad "example" to foist on people by not also following the established pattern of freeing the storage , so I'll make a modification for this shortly. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, March 12, 2010 11:47 PM To: m3commit at elegosoft.com; Coleburn, Randy Subject: RE: [M3commit] CVS Update: cm3 This change is suspicious. I think it leaks. Please see http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/os/WIN32/FSWin32.m3?rev=1.7;content-type=text%2Fplain for the pattern that is usually used. Granted, given the conservativeness of the collector -- it finds stuff in registers and the stack without a need for metadata telling it where the points are, and the fact that we no longer have VM-synchronization, I don't understand the need for the usual pattern. Maybe because it is willing to move stuff? Therefore a non-traced copy is made for C that won't ever move? - Jay > Date: Sat, 13 Mar 2010 02:11:24 +0000 > To: m3commit at elegosoft.com > From: rcoleburn at elego.de > Subject: [M3commit] CVS Update: cm3 > > CVSROOT: /usr/cvs > Changes by: rcoleburn at birch. 10/03/13 02:11:24 > > Modified files: > cm3/examples/calling-c-win32/src/: OK.m3 > > Log message: > M3toC.TtoS has been renamed to M3toC.SharedTtoS, so make adjustments to Ok.m3 so that it will compile.--R.Coleburn > ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 13 07:36:25 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 13 Mar 2010 06:36:25 +0000 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20100313011125.34CF12474001@birch.elegosoft.com>, , Message-ID: Bad examples beget bad code. Bad (and good) examples get copied into real code. Don't make them. It wasn't a rename, it was I believe a semantic change accompanied by a rename. Better to not compile than compile and be incorrect. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Sat, 13 Mar 2010 00:57:00 -0500 Subject: Re: [M3devel] [M3commit] CVS Update: cm3 Jay: This is example code. The program terminates after the message box is displayed. There is no further processing. I did not write this example; I merely fixed it to compile again after someone changed the names of the procedures in the M3toC interface. Nevertheless, it is probably a bad ?example? to foist on people by not also following the established pattern of freeing the storage , so I?ll make a modification for this shortly. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, March 12, 2010 11:47 PM To: m3commit at elegosoft.com; Coleburn, Randy Subject: RE: [M3commit] CVS Update: cm3 This change is suspicious. I think it leaks. Please see http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/os/WIN32/FSWin32.m3?rev=1.7;content-type=text%2Fplain for the pattern that is usually used. Granted, given the conservativeness of the collector -- it finds stuff in registers and the stack without a need for metadata telling it where the points are, and the fact that we no longer have VM-synchronization, I don't understand the need for the usual pattern. Maybe because it is willing to move stuff? Therefore a non-traced copy is made for C that won't ever move? - Jay > Date: Sat, 13 Mar 2010 02:11:24 +0000 > To: m3commit at elegosoft.com > From: rcoleburn at elego.de > Subject: [M3commit] CVS Update: cm3 > > CVSROOT: /usr/cvs > Changes by: rcoleburn at birch. 10/03/13 02:11:24 > > Modified files: > cm3/examples/calling-c-win32/src/: OK.m3 > > Log message: > M3toC.TtoS has been renamed to M3toC.SharedTtoS, so make adjustments to Ok.m3 so that it will compile.--R.Coleburn > CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Sat Mar 13 07:49:55 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sat, 13 Mar 2010 01:49:55 -0500 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20100313011125.34CF12474001@birch.elegosoft.com>, , Message-ID: Jay: I agree "bad examples beget bad code", but I don't need to be lectured. I didn't create this example code. The example code is from the "Reactor v4.1 days and was produced by Critical Mass, Inc." Regardless of how the M3toC interface has changed since that time, it is obvious that the example code was not updated to be consistent with the changes. Thus, we had an example that failed to even compile, and that's not good either, especially since CM3IDE creates a copy of the example package in your private repository and lets you try to build and run it. I have repaired this example so that it compiles and runs. I've also made the change to free the memory to serve as a better exemplar of proper practice, even though it really doesn't affect the running of the program since the program terminates after the message box is closed and no further processing is done. I agree it was bad style in the first place on the part of the folks at Critical Mass. Now it is better. You will probably find other bad style examples, so if you want to fix them, be my guest. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Saturday, March 13, 2010 1:36 AM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] [M3commit] CVS Update: cm3 Bad examples beget bad code. Bad (and good) examples get copied into real code. Don't make them. It wasn't a rename, it was I believe a semantic change accompanied by a rename. Better to not compile than compile and be incorrect. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Sat, 13 Mar 2010 00:57:00 -0500 Subject: Re: [M3devel] [M3commit] CVS Update: cm3 Jay: This is example code. The program terminates after the message box is displayed. There is no further processing. I did not write this example; I merely fixed it to compile again after someone changed the names of the procedures in the M3toC interface. Nevertheless, it is probably a bad "example" to foist on people by not also following the established pattern of freeing the storage , so I'll make a modification for this shortly. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, March 12, 2010 11:47 PM To: m3commit at elegosoft.com; Coleburn, Randy Subject: RE: [M3commit] CVS Update: cm3 This change is suspicious. I think it leaks. Please see http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/os/WIN32/FSWin32.m3?rev=1.7;content-type=text%2Fplain for the pattern that is usually used. Granted, given the conservativeness of the collector -- it finds stuff in registers and the stack without a need for metadata telling it where the points are, and the fact that we no longer have VM-synchronization, I don't understand the need for the usual pattern. Maybe because it is willing to move stuff? Therefore a non-traced copy is made for C that won't ever move? - Jay > Date: Sat, 13 Mar 2010 02:11:24 +0000 > To: m3commit at elegosoft.com > From: rcoleburn at elego.de > Subject: [M3commit] CVS Update: cm3 > > CVSROOT: /usr/cvs > Changes by: rcoleburn at birch. 10/03/13 02:11:24 > > Modified files: > cm3/examples/calling-c-win32/src/: OK.m3 > > Log message: > M3toC.TtoS has been renamed to M3toC.SharedTtoS, so make adjustments to Ok.m3 so that it will compile.--R.Coleburn > ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 13 11:19:21 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 13 Mar 2010 10:19:21 +0000 Subject: [M3devel] comparisons vs. subranges Message-ID: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 13 11:49:37 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 13 Mar 2010 10:49:37 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: Message-ID: I might be able to do this. Give me a day or so.. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu; m3devel at elegosoft.com Date: Sat, 13 Mar 2010 10:19:21 +0000 Subject: [M3devel] comparisons vs. subranges <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Sat Mar 13 19:29:24 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Sat, 13 Mar 2010 13:29:24 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: Message-ID: <20100313182924.GA14329@topoi.pooq.com> On Sat, Mar 13, 2010 at 10:19:21AM +0000, Jay K wrote: > > <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; > <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; > > > > > Seems to me, the front end should notice these. > > The first should always be TRUE. > > And possibly, possibly warn. > > The second should always be FALSE. > > And possibly, possibly warn. > > > > "Generic" programming often hits this sort of thing though, a good reason not to warn. > > Programmer might also be working with existing code and have changed INTEGER to CARDINAL. > > Or be defending against future maintainers changing CARDINAL to INTEGER. Wasn't there a discussion a while ago about subranges out-of-bounds not being a safety problem? (Or was it arithmetic overflow?) This optimisation might well cause a quite hard-to-find bug if we don't guarantee subrange integrity. -- hendrik From jay.krell at cornell.edu Sat Mar 13 17:57:27 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 13 Mar 2010 16:57:27 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <20100313182924.GA14329@topoi.pooq.com> References: , <20100313182924.GA14329@topoi.pooq.com> Message-ID: Integer overflow is not a safety problem. That was the news (to me). Subranges do need to be enforced, at certain points, for their to be safety. This change doesn't change that. (Compiler bugs break safety, as always.) - Jay > Date: Sat, 13 Mar 2010 13:29:24 -0500 > From: hendrik at topoi.pooq.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] comparisons vs. subranges > > On Sat, Mar 13, 2010 at 10:19:21AM +0000, Jay K wrote: > > > > <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; > > <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; > > > > > > > > > > Seems to me, the front end should notice these. > > > > The first should always be TRUE. > > > > And possibly, possibly warn. > > > > The second should always be FALSE. > > > > And possibly, possibly warn. > > > > > > > > "Generic" programming often hits this sort of thing though, a good reason not to warn. > > > > Programmer might also be working with existing code and have changed INTEGER to CARDINAL. > > > > Or be defending against future maintainers changing CARDINAL to INTEGER. > > Wasn't there a discussion a while ago about subranges out-of-bounds not > being a safety problem? (Or was it arithmetic overflow?) This > optimisation might well cause a quite hard-to-find bug if we don't > guarantee subrange integrity. > > -- hendrik -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sat Mar 13 19:03:18 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sat, 13 Mar 2010 13:03:18 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: Message-ID: Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: > <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; > <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; > > > Seems to me, the front end should notice these. > The first should always be TRUE. > And possibly, possibly warn. > The second should always be FALSE. > And possibly, possibly warn. > > "Generic" programming often hits this sort of thing though, a good reason not to warn. > Programmer might also be working with existing code and have changed INTEGER to CARDINAL. > Or be defending against future maintainers changing CARDINAL to INTEGER. > > > The backend isn't give enough information, because CARDINAL = INTEGER as far > as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. > The only way to get unsigned types is to use ADDRESS from what I see. > > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 13 19:26:34 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 13 Mar 2010 18:26:34 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: , Message-ID: Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. What is the point of these functions Fold? Why does it have these various "cost" computations? Granted, they are rough. Should we beef up m3middle? m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. m3back is correct or extremely close to correct as far as I know. It is missing atomic operations for 8, 16, 64 bit operands. (PPC32 seems unable to support 64bit atomics btw.) I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. - Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 13:03:18 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] comparisons vs. subranges Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sat Mar 13 20:23:33 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sat, 13 Mar 2010 14:23:33 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: , Message-ID: On 13 Mar 2010, at 13:26, Jay K wrote: > Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. > > > What is the point of these functions Fold? So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. > Why does it have these various "cost" computations? Which cost computations are you referring to? > Granted, they are rough. > > > Should we beef up m3middle? That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. > m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. > It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. > It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. Designing a decent optimising middle-end takes significant time and effort, and will require more thought. > m3back is correct or extremely close to correct as far as I know. > It is missing atomic operations for 8, 16, 64 bit operands. > (PPC32 seems unable to support 64bit atomics btw.) Correct. > I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. > > > I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Sat, 13 Mar 2010 13:03:18 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] comparisons vs. subranges > > Jay, > > I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 13 Mar 2010, at 05:19, Jay K wrote: > > <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; > <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; > > > Seems to me, the front end should notice these. > The first should always be TRUE. > And possibly, possibly warn. > The second should always be FALSE. > And possibly, possibly warn. > > "Generic" programming often hits this sort of thing though, a good reason not to warn. > Programmer might also be working with existing code and have changed INTEGER to CARDINAL. > Or be defending against future maintainers changing CARDINAL to INTEGER. > > > The backend isn't give enough information, because CARDINAL = INTEGER as far > as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. > The only way to get unsigned types is to use ADDRESS from what I see. > > > - Jay > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sat Mar 13 20:45:29 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sat, 13 Mar 2010 14:45:29 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: , Message-ID: <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Jay, I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 14:23, Tony Hosking wrote: > On 13 Mar 2010, at 13:26, Jay K wrote: > >> Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. >> >> >> What is the point of these functions Fold? > > So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. > >> Why does it have these various "cost" computations? > > Which cost computations are you referring to? > >> Granted, they are rough. >> >> >> Should we beef up m3middle? > > That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. > >> m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. >> It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. >> It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. > > Designing a decent optimising middle-end takes significant time and effort, and will require more thought. > >> m3back is correct or extremely close to correct as far as I know. >> It is missing atomic operations for 8, 16, 64 bit operands. >> (PPC32 seems unable to support 64bit atomics btw.) > > Correct. > >> I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. >> >> >> I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. >> >> >> - Jay >> >> >> From: hosking at cs.purdue.edu >> Date: Sat, 13 Mar 2010 13:03:18 -0500 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] comparisons vs. subranges >> >> Jay, >> >> I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. >> >> Antony Hosking | Associate Professor | Computer Science | Purdue University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >> On 13 Mar 2010, at 05:19, Jay K wrote: >> >> <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; >> <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; >> >> >> Seems to me, the front end should notice these. >> The first should always be TRUE. >> And possibly, possibly warn. >> The second should always be FALSE. >> And possibly, possibly warn. >> >> "Generic" programming often hits this sort of thing though, a good reason not to warn. >> Programmer might also be working with existing code and have changed INTEGER to CARDINAL. >> Or be defending against future maintainers changing CARDINAL to INTEGER. >> >> >> The backend isn't give enough information, because CARDINAL = INTEGER as far >> as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. >> The only way to get unsigned types is to use ADDRESS from what I see. >> >> >> - Jay >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmuysers at hotmail.com Sat Mar 13 23:20:07 2010 From: dmuysers at hotmail.com (Dirk Muysers) Date: Sat, 13 Mar 2010 23:20:07 +0100 Subject: [M3devel] bandwith Message-ID: I don't know if other people have the same feeling, but I think that Messrs. Krell and Hosking should keep their polemics to themselves and let the community partake only in the result of a constructive consensus. As a USER of the m3 compiler on windows, I find the recent developments scary, so I prefer to stick to my installed rather old release (5.3.1 if memory serves, BTW the version number should figure at a prominent place in every release.) I am afraid that M3 is going the way of fubar and I am slowly loosing interest. To make it work for all these different platforms seems a herculean task and rather pointless endeavour. Why not use all that energy to try a radically disruptive strategy such as a platform agnostic intermediate form targeting a JIT. Let me apologize in advance to the whole community for the present fit of an old men's anger. d. muysers -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Sun Mar 14 02:54:41 2010 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sun, 14 Mar 2010 01:54:41 +0000 (GMT) Subject: [M3devel] bandwith In-Reply-To: Message-ID: <562452.30843.qm@web23608.mail.ird.yahoo.com> Hi all: I think is an interesting topic, however I disagree with you in the old version mentioned because since then CM3 has evolved a lot in threading, garbage collection and gcc based backend packages, just to name a few. I however seem to recall some discussions, but unfortunately there wasn't sometimes an ending point (i.e my discussion), however without more participation it really becomes boring at one point I expect to be fully honest I do have fun compiling things but probably the community lacks more understanding of the system itself or of the current issues to work in that which needs to be done. Track system by itself is great but probably we haven't used yet in its full potential and that's the reason I waited for so long to report things that would be useful before in ticket trac system (nor I did have the idea to ask you). Current situation in this matter is going better I think. Therefore there could be plenty of opportunities of things to work on in a fashionable way if we may. I don't know but I think the current stability should be the worrisome and the history of this could go back to C interfaces and son on. I haven't asked yet where are the Unix INTERFACE constants and members which are need for packages that before where compiling that use this low-level style like browsers webcard, postcard, Deckscape and Webscape and m3-mail packages and m3-lectern which I could get to compile some time before this Unix changes. However I know that some bugs they had I expect to be gone because this changes brought many other improvements to overall programming environment. Besides this, software engineering tools could be made aware of the project like this one done for pm3: http://libresoft.dat.escet.urjc.es/cvsanal/pm3-cvs/index.php?menu=Index (the manual of it on http://gsyc.es/~carlosgc/files/cvsanaly.pdf software on http://tools.libresoft.es/cvsanaly?rev=1248394389) I know there is also a Modula-3 aware re engineering tools it was done by Peter Klien at Aachen University I know there was supossed to be a working prototype at some point available for download http://pi.informatik.uni-siegen.de/stt/23_3/05_Dissertationen/klein.pdf This kind of tools could be managed to get integrated with M3clipse plugin work done by Bert Laverman which just lacks semantic analysis, and cm3ide which may bundle cvsup and or dcvs as a way of distributed software configuration management and development environment perhaps with Elego Compact (and don't forget Obliq packages, I remember a Professor told me that it competed at some point with JavaScript to be scripting language for the web. It has even type inference algorithm now so who knows why isn't used in our community, I know I will try to get into it but it needs time; there are some examples in Professor's courses that could help to this, besides other things of M3 system level demo applications too) Thanks for your comments I hope this helps a bit --- El s?b, 13/3/10, Dirk Muysers escribi?: > De: Dirk Muysers > Asunto: [M3devel] bandwith > Para: m3devel at elegosoft.com > Fecha: s?bado, 13 de marzo, 2010 17:20 > > > > > > > > I don't know if other > people have the same feeling, > but I think that Messrs. Krell and Hosking > should keep their polemics to > themselves and let the > community partake only > in the result of > a constructive consensus. > > > As a USER of the m3 > compiler on windows, I > find the recent developments scary, so I prefer > to > stick to my installed > rather old release (5.3.1 if > memory serves, BTW the version number should > figure at a prominent > place in every > release.) I am afraid that M3 is going the way of fubar and > I > am slowly loosing > interest. > > To make it work for all > these different platforms > seems a herculean task > and rather > pointless > endeavour. Why not use all > that energy to > try a radically > disruptive strategy such as > a platform > agnostic intermediate form > targeting a > JIT. > > Let > me apologize in advance to the whole > community for the present fit of an old men's > anger. > d. > muysers > From jay.krell at cornell.edu Sun Mar 14 04:52:15 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 03:52:15 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: Hm. You mean, like I'm not supposed to be able to say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (a < b) => IO.Put("true\n"); | (a > b) => IO.Put("false\n"); END; but I can say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); END; (I'd like to be able to say FIRST(a), etc., but I don't think it is allowed.) "constants" kind of seem like an optimization themselves. Other than efficiency concerns, they could all be variables initialized by module initializers, that the frontend doesn't let you write to. Including de-optimizations like runtime allocation/sizing of arrays based on const/non-const sizes. Look at how I changed const to var already in m3core/unix, and then had to change CASE to if/elseif ladders. It's a minor matter of what the language lets you say, among various basically equivalent forms. The compiler could allow case to target non-constants. It'd just mainly be slower. There is the matter of CASE arms can't overlap, where if/elseif ladders can. Because CASE is conceptually compared all at once where if/elseif is sequential. Ok. I see a few options at least. m3middle could look basically like m3front, but with various checks like type checking removed and assumed true. This would be an arduous task of duplication. M3CG.i3 Var and Target.Int could gain the following fields (or methods): min_valid := FALSE; max_valid := FALSE; min := TInt.Zero; max := TInt.Zero; m3front could fill them in a few cases. Initially none, but then a few as they are discovered to be easy, correct, efficient. For constants, min = max value. For subranges, min/max come the type. The backends could use them (if valid) and possibly transform them. e.g. MIN(a,b) with valid min/max yields an expression where min is the min of the two mins, max is the min of the two maxes. MIN([0..1],[2..3]) < 2 => TRUE. Probably more correct would be: PROCEDURE declare_enum (xx: T; t: TypeUID; n_elts: INTEGER; s: BitSize) = PROCEDURE declare_subrange (xx: T; t, domain: TypeUID; READONLY min, max: Target.Int; s: BitSize) => already exist and then PROCEDURE load (xx: T; v: Var; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_indirect (xx: T; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_integer (xx: T; t: IType; READONLY i: Target.Int) = PROCEDURE load_ordered (xx: T; t: MType; u: ZType; order: MemoryOrder) = etc. should all take an optional TypeUID. The second proposal I've thought a bit more about and it seems very easy. The last idea seems cleaner, but with possibly signifiant downside in that e.g. Word.T is not a subrange. There's also some unclarity with respect to Word.LT(expression, -1); -1 is a valid Word interpreted as a large number. One would want to be careful not to break that. I believe this is related to what you were saying, where operations are typed, not values. It could be that every operation needs min/max or typeuid (pairs of them for binary operations). Can we take a stab at something like this? This might also serve to replace some of what m3back does with constants already, since constants would fall out of subranges of length 1. Thanks, -Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 14:45:29 -0500 To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com; jay.krell at cornell.edu Subject: Re: [M3devel] comparisons vs. subranges Jay, I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 14:23, Tony Hosking wrote: On 13 Mar 2010, at 13:26, Jay K wrote: Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. What is the point of these functions Fold? So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. Why does it have these various "cost" computations? Which cost computations are you referring to? Granted, they are rough. Should we beef up m3middle? That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. Designing a decent optimising middle-end takes significant time and effort, and will require more thought. m3back is correct or extremely close to correct as far as I know. It is missing atomic operations for 8, 16, 64 bit operands. (PPC32 seems unable to support 64bit atomics btw.) Correct. I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. - Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 13:03:18 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] comparisons vs. subranges Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 05:31:58 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 04:31:58 +0000 Subject: [M3devel] bandwith In-Reply-To: <562452.30843.qm@web23608.mail.ird.yahoo.com> References: , <562452.30843.qm@web23608.mail.ird.yahoo.com> Message-ID: Arg. Change does not equal instability. Status quo does not equal better. Do you want 64bit LONGINT? Maybe, maybe not. Do you want every integer comparison to needlessly use a temporary on the stack and extra instructions? Maybe, maybe not. Do you want the tables in hand.c statically linked? Maybe, maybe not. They caused quite a headache albeit briefly, when we weren't statically linking them. Standalone apps worked fine, dynamically linked exhibited odd behavior. The odd behavior was "just" gui drawn incorrectly, but was actually indicative of deep problems. Do you want dead buggy code sitting around, waiting to be silently used as m3front changes? (e.g. some forms of extract look wrong). A giant lock in ThreadWin32? vs. the simple pattern documented on the web and what Java uses? Granted, I removed the threadpool, maybe too much simplicity and not enough efficiency? (But have you noticed how much ThreadWin32.m3 did churn through the years anyway?) Can you point to actual bugs? Granted, I don't have specifics in many cases either, such as the bad perf of the giant lock, but everyone knows about them. The system builds itself. I probably rest too much on that, granted. m3-sys/m3tests does pretty well (some stuff does need looking at). I try to add more test coverage when I make changes, but I am lazy, granted. Some non-zero level of progress is presumably desired. The Unix interface was painful to maintain and buggy and tended toward unsafety but claiming safety. It is much improved now. Much safer, much easier to port. A little less efficient. Sort off less amenable to cross building, though really hand.c and lack of an assembler and linker are enough to kill cross builds. The system has never as far I know been really cross buildable, without gcc/binutils support -- has always needed an assembler and linker, and usually needed a C compiler. (NT386 doesn't need an assembler). Just because interface Unix allowed for compiling more code in the past did not make it better or more correct. In fact possibly the opposite. Granted, we should try have it all -- correctness, features, maintability, efficiency. There was a lot of cruft in there getting copied from port to port without being revisited. Stuff having to do with Linux 1.x was in various ports. Every major version of FreeBSD implied extra work. We had FreeBSD, FreeBSD2, FreeBSD3, FreeBSD4. (There are still problems here actually, FreeBSD7 64bit I believe broke the ABI in networking, and we still have the related structs I think -- there is still bad stuff in interface Unix, just a lot less now. I still want to whittle it down more.) There is still code in the system that redeclares struct tm incorrectly, attempting to workaround interface Unix deficiency, when in fact interface Unix was correctly limited (the fields at the end, and perhaps the order of the fields). In the web browsers as I recall (which never seemed to work much in the first place, but they did/do build and come up). > > agnostic intermediate form targeting a JIT. Other than NT386, our portability rests on gcc and Posix, which are very portable with very little effort. I haven't seen any JIT that is as portable. For example there are a few systems we can't run Hudson on because it uses Java. e.g. MIPS64_OPENBSD, PPC32_OPENBSD I would actually like to generate C to gain even more portability. Granted, JIT has the nice property of target-independent distributable binaries. "Package building" matrix could be cut down. Generating C can get you target-independent source distributions, as most things use. (Granted, compilers are somewhat special, except that the world long ago got past not having a C compiler). Targeting a JIT will probably be way more work than anyone has put into this system in over a decade. Still, not a bad idea. It lets you reuse other people's code generators similarly to how we reuse gcc. Java has been made portable to any processor, running Linux. But not yet to other operating systems. I think without too much work, we could use our exiting IL that parse.c reads. It'd just need more context in places, to avoid a need for globals. You could just write an interpreter. It wouldn't be fast, but it would gain more portability probably. Probably not the best idea though. Also, the Java JIT, as one example, is among the more portable, and among the more difficult to target. It doesn't model a CPU as you might expect. It models more so the Java programming language. Which includes non-optional safety. You'd have better luck probably with mono/CLR, not sure as to the level of portability. Mono is pretty widely ported though. Really though, gcc+Posix provides a huge amount of portability with very little effort on our part. The larger work is really obtaining, housing, powering, installing the various systems. If you go the C/portable-JIT route, what you really do is you just stop testing the various targets, for better and worse, probably perfectly ok. Just as nobody tests hello world on every system. They just know it works because it is so portable. > webcard, postcard, Deckscape and Webscape and m3-mail packages and m3-lectern They don't compile? > > rather old release (5.3.1 if memory serves, You can keep using it? Maybe compare that version to current and see if there are particular problems? > > BTW the version number should figure at a prominent place in every release. cm3 -version? As to where Tony and I should argue, well, that's not a generally answerable question. I find that a conversation involving just two people not ideal, no matter who they are. Being under outside scrutiny increases civility, sometimes add more than two opinions, etc. I grant that m3 is such a small community that we don't have an m3-users vs. m3-devel group, and m3-devel is woefully small, and the two groups a bit inappropriately mixed. - Jay > Date: Sun, 14 Mar 2010 01:54:41 +0000 > From: dabenavidesd at yahoo.es > To: m3devel at elegosoft.com; dmuysers at hotmail.com > Subject: Re: [M3devel] bandwith > > Hi all: > I think is an interesting topic, however I disagree with you in the old version mentioned because since then CM3 has evolved a lot in threading, garbage collection and gcc based backend packages, just to name a few. > I however seem to recall some discussions, but unfortunately there wasn't sometimes an ending point (i.e my discussion), however without more participation it really becomes boring at one point I expect to be fully honest I do have fun compiling things but probably the community lacks more understanding of the system itself or of the current issues to work in that which needs to be done. Track system by itself is great but probably we haven't used yet in its full potential and that's the reason I waited for so long to report things that would be useful before in ticket trac system (nor I did have the idea to ask you). Current situation in this matter is going better I think. > Therefore there could be plenty of opportunities of things to work on in a fashionable way if we may. > I don't know but I think the current stability should be the worrisome and the history of this could go back to C interfaces and son on. I haven't asked yet where are the Unix INTERFACE constants and members which are need for packages that before where compiling that use this low-level style like browsers webcard, postcard, Deckscape and Webscape and m3-mail packages and m3-lectern which I could get to compile some time before this Unix changes. > However I know that some bugs they had I expect to be gone because this changes brought many other improvements to overall programming environment. > Besides this, software engineering tools could be made aware of the project like this one done for pm3: > http://libresoft.dat.escet.urjc.es/cvsanal/pm3-cvs/index.php?menu=Index > (the manual of it on http://gsyc.es/~carlosgc/files/cvsanaly.pdf software on http://tools.libresoft.es/cvsanaly?rev=1248394389) > I know there is also a Modula-3 aware re engineering tools it was done by Peter Klien at Aachen University I know there was supossed to be a working prototype at some point available for download > http://pi.informatik.uni-siegen.de/stt/23_3/05_Dissertationen/klein.pdf > This kind of tools could be managed to get integrated with M3clipse plugin work done by Bert Laverman which just lacks semantic analysis, and cm3ide which may bundle cvsup and or dcvs as a way of distributed software configuration management and development environment perhaps with Elego Compact (and don't forget Obliq packages, I remember a Professor told me that it competed at some point with JavaScript to be scripting language for the web. It has even type inference algorithm now so who knows why isn't used in our community, I know I will try to get into it but it needs time; there are some examples in Professor's courses that could help to this, besides other things of M3 system level demo applications too) > Thanks for your comments I hope this helps a bit > > > --- El s?b, 13/3/10, Dirk Muysers escribi?: > > > De: Dirk Muysers > > Asunto: [M3devel] bandwith > > Para: m3devel at elegosoft.com > > Fecha: s?bado, 13 de marzo, 2010 17:20 > > > > > > > > > > > > > > > > I don't know if other > > people have the same feeling, > > but I think that Messrs. Krell and Hosking > > should keep their polemics to > > themselves and let the > > community partake only > > in the result of > > a constructive consensus. > > > > > > As a USER of the m3 > > compiler on windows, I > > find the recent developments scary, so I prefer > > to > > stick to my installed > > rather old release (5.3.1 if > > memory serves, BTW the version number should > > figure at a prominent > > place in every > > release.) I am afraid that M3 is going the way of fubar and > > I > > am slowly loosing > > interest. > > > > To make it work for all > > these different platforms > > seems a herculean task > > and rather > > pointless > > endeavour. Why not use all > > that energy to > > try a radically > > disruptive strategy such as > > a platform > > agnostic intermediate form > > targeting a > > JIT. > > > > Let > > me apologize in advance to the whole > > community for the present fit of an old men's > > anger. > > d. > > muysers > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 06:01:24 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 05:01:24 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: Hm. How about elsewhere m3front? Optimizing middle end not necessarily a different package? (esp. given that m3front is already big enough to afford being organized into directories: m3front/src/middle, m3front/src/target-independent-optimizations?) Maybe elsewhere in the two files I changed? Prep or PrepBR or Compile instead of Fold? I don't understand this Prep vs. PrepBR business. I found out that "BR" stands for branch, but that doesn't explain it me. Also PrepBR and Compile look very similar. I also noticed that Fold gets called twice (I had some RTIO in it) but didn't look into why. Surely doing it in PrepBR or Compile is correct or CG.m3 is correct, reasonable, efficient, maintainable? I can see why Fold would be wrong -- allowing code to compile, with a reasonable meaning, but that isn't supposed to. Or in CG.m3, which is conceptually "the bottom" of m3front, any two adjacent pieces can be considered merged into one layer. I'll look at both of those options later. I think "subrange analysis" if done enough might yield efficiencies in real code. Though m3front might already do enough of it -- e.g. the fact that you can't modify a FOR loop index affords some efficiencies, that are probably already taken into account. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 03:52:15 +0000 Hm. You mean, like I'm not supposed to be able to say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (a < b) => IO.Put("true\n"); | (a > b) => IO.Put("false\n"); END; but I can say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); END; (I'd like to be able to say FIRST(a), etc., but I don't think it is allowed.) "constants" kind of seem like an optimization themselves. Other than efficiency concerns, they could all be variables initialized by module initializers, that the frontend doesn't let you write to. Including de-optimizations like runtime allocation/sizing of arrays based on const/non-const sizes. Look at how I changed const to var already in m3core/unix, and then had to change CASE to if/elseif ladders. It's a minor matter of what the language lets you say, among various basically equivalent forms. The compiler could allow case to target non-constants. It'd just mainly be slower. There is the matter of CASE arms can't overlap, where if/elseif ladders can. Because CASE is conceptually compared all at once where if/elseif is sequential. Ok. I see a few options at least. m3middle could look basically like m3front, but with various checks like type checking removed and assumed true. This would be an arduous task of duplication. M3CG.i3 Var and Target.Int could gain the following fields (or methods): min_valid := FALSE; max_valid := FALSE; min := TInt.Zero; max := TInt.Zero; m3front could fill them in a few cases. Initially none, but then a few as they are discovered to be easy, correct, efficient. For constants, min = max value. For subranges, min/max come the type. The backends could use them (if valid) and possibly transform them. e.g. MIN(a,b) with valid min/max yields an expression where min is the min of the two mins, max is the min of the two maxes. MIN([0..1],[2..3]) < 2 => TRUE. Probably more correct would be: PROCEDURE declare_enum (xx: T; t: TypeUID; n_elts: INTEGER; s: BitSize) = PROCEDURE declare_subrange (xx: T; t, domain: TypeUID; READONLY min, max: Target.Int; s: BitSize) => already exist and then PROCEDURE load (xx: T; v: Var; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_indirect (xx: T; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_integer (xx: T; t: IType; READONLY i: Target.Int) = PROCEDURE load_ordered (xx: T; t: MType; u: ZType; order: MemoryOrder) = etc. should all take an optional TypeUID. The second proposal I've thought a bit more about and it seems very easy. The last idea seems cleaner, but with possibly signifiant downside in that e.g. Word.T is not a subrange. There's also some unclarity with respect to Word.LT(expression, -1); -1 is a valid Word interpreted as a large number. One would want to be careful not to break that. I believe this is related to what you were saying, where operations are typed, not values. It could be that every operation needs min/max or typeuid (pairs of them for binary operations). Can we take a stab at something like this? This might also serve to replace some of what m3back does with constants already, since constants would fall out of subranges of length 1. Thanks, -Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 14:45:29 -0500 To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com; jay.krell at cornell.edu Subject: Re: [M3devel] comparisons vs. subranges Jay, I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 14:23, Tony Hosking wrote: On 13 Mar 2010, at 13:26, Jay K wrote: Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. What is the point of these functions Fold? So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. Why does it have these various "cost" computations? Which cost computations are you referring to? Granted, they are rough. Should we beef up m3middle? That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. Designing a decent optimising middle-end takes significant time and effort, and will require more thought. m3back is correct or extremely close to correct as far as I know. It is missing atomic operations for 8, 16, 64 bit operands. (PPC32 seems unable to support 64bit atomics btw.) Correct. I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. - Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 13:03:18 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] comparisons vs. subranges Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 06:15:22 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 05:15:22 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: > Prep vs. PrepBR vs. Compile Expr.i3 documents it. (*** phase 4 ****) (* Expressions are compiled in two steps: Prep: emit any code that includes branchs or stores Compile: emit the rest of the code *) PROCEDURE Prep (t: T); PROCEDURE Compile (t: T); (* emits code to evaluate the expression onto the top of stack *) PROCEDURE PrepLValue (t: T; traced: BOOLEAN); PROCEDURE CompileLValue (t: T; traced: BOOLEAN); (* emits code to evaluate 't's L-value into s0.A. *) PROCEDURE CompileAddress (t: T; traced: BOOLEAN); (* emits code to evaluate 't's byte address onto the top of stack. Use PrepLValue to prep these expressions. *) PROCEDURE PrepBranch (t: T; true, false: CG.Label; freq: CG.Frequency); PROCEDURE CompileBranch (t: T; true, false: CG.Label; freq: CG.Frequency); (* emits code to evaluate the expression and conditionally branch to 'true' or 'false' depending on its boolean value. 'freq' is the estimated frequency that the specified branch will be taken. *) - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 05:01:24 +0000 Hm. How about elsewhere m3front? Optimizing middle end not necessarily a different package? (esp. given that m3front is already big enough to afford being organized into directories: m3front/src/middle, m3front/src/target-independent-optimizations?) Maybe elsewhere in the two files I changed? Prep or PrepBR or Compile instead of Fold? I don't understand this Prep vs. PrepBR business. I found out that "BR" stands for branch, but that doesn't explain it me. Also PrepBR and Compile look very similar. I also noticed that Fold gets called twice (I had some RTIO in it) but didn't look into why. Surely doing it in PrepBR or Compile is correct or CG.m3 is correct, reasonable, efficient, maintainable? I can see why Fold would be wrong -- allowing code to compile, with a reasonable meaning, but that isn't supposed to. Or in CG.m3, which is conceptually "the bottom" of m3front, any two adjacent pieces can be considered merged into one layer. I'll look at both of those options later. I think "subrange analysis" if done enough might yield efficiencies in real code. Though m3front might already do enough of it -- e.g. the fact that you can't modify a FOR loop index affords some efficiencies, that are probably already taken into account. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 03:52:15 +0000 Hm. You mean, like I'm not supposed to be able to say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (a < b) => IO.Put("true\n"); | (a > b) => IO.Put("false\n"); END; but I can say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); END; (I'd like to be able to say FIRST(a), etc., but I don't think it is allowed.) "constants" kind of seem like an optimization themselves. Other than efficiency concerns, they could all be variables initialized by module initializers, that the frontend doesn't let you write to. Including de-optimizations like runtime allocation/sizing of arrays based on const/non-const sizes. Look at how I changed const to var already in m3core/unix, and then had to change CASE to if/elseif ladders. It's a minor matter of what the language lets you say, among various basically equivalent forms. The compiler could allow case to target non-constants. It'd just mainly be slower. There is the matter of CASE arms can't overlap, where if/elseif ladders can. Because CASE is conceptually compared all at once where if/elseif is sequential. Ok. I see a few options at least. m3middle could look basically like m3front, but with various checks like type checking removed and assumed true. This would be an arduous task of duplication. M3CG.i3 Var and Target.Int could gain the following fields (or methods): min_valid := FALSE; max_valid := FALSE; min := TInt.Zero; max := TInt.Zero; m3front could fill them in a few cases. Initially none, but then a few as they are discovered to be easy, correct, efficient. For constants, min = max value. For subranges, min/max come the type. The backends could use them (if valid) and possibly transform them. e.g. MIN(a,b) with valid min/max yields an expression where min is the min of the two mins, max is the min of the two maxes. MIN([0..1],[2..3]) < 2 => TRUE. Probably more correct would be: PROCEDURE declare_enum (xx: T; t: TypeUID; n_elts: INTEGER; s: BitSize) = PROCEDURE declare_subrange (xx: T; t, domain: TypeUID; READONLY min, max: Target.Int; s: BitSize) => already exist and then PROCEDURE load (xx: T; v: Var; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_indirect (xx: T; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_integer (xx: T; t: IType; READONLY i: Target.Int) = PROCEDURE load_ordered (xx: T; t: MType; u: ZType; order: MemoryOrder) = etc. should all take an optional TypeUID. The second proposal I've thought a bit more about and it seems very easy. The last idea seems cleaner, but with possibly signifiant downside in that e.g. Word.T is not a subrange. There's also some unclarity with respect to Word.LT(expression, -1); -1 is a valid Word interpreted as a large number. One would want to be careful not to break that. I believe this is related to what you were saying, where operations are typed, not values. It could be that every operation needs min/max or typeuid (pairs of them for binary operations). Can we take a stab at something like this? This might also serve to replace some of what m3back does with constants already, since constants would fall out of subranges of length 1. Thanks, -Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 14:45:29 -0500 To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com; jay.krell at cornell.edu Subject: Re: [M3devel] comparisons vs. subranges Jay, I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 14:23, Tony Hosking wrote: On 13 Mar 2010, at 13:26, Jay K wrote: Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. What is the point of these functions Fold? So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. Why does it have these various "cost" computations? Which cost computations are you referring to? Granted, they are rough. Should we beef up m3middle? That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. Designing a decent optimising middle-end takes significant time and effort, and will require more thought. m3back is correct or extremely close to correct as far as I know. It is missing atomic operations for 8, 16, 64 bit operands. (PPC32 seems unable to support 64bit atomics btw.) Correct. I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. - Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 13:03:18 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] comparisons vs. subranges Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 11:43:57 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 10:43:57 +0000 Subject: [M3devel] comparing different types/signedness Message-ID: Tony, is it deliberate in m3front/src/misc/cg.m3 procedure compare that we are often asked to compare different types, even different signedness? It seems a little unusual. I think it really might help to extend TInt to 9 or more bytes so that any pair of values we "really" encounter can be extended to a common type/precision. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 13:36:50 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 12:36:50 +0000 Subject: [M3devel] comparing different types/signedness In-Reply-To: References: Message-ID: I got through my problem here, not a big deal. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu; m3devel at elegosoft.com Date: Sun, 14 Mar 2010 10:43:57 +0000 Subject: [M3devel] comparing different types/signedness Tony, is it deliberate in m3front/src/misc/cg.m3 procedure compare that we are often asked to compare different types, even different signedness? It seems a little unusual. I think it really might help to extend TInt to 9 or more bytes so that any pair of values we "really" encounter can be extended to a common type/precision. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 13:43:02 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 12:43:02 +0000 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: Tony how about the attached? It achieves the "same" thing as before. CG is about the lowest level in m3front, therefore akin to any middle end below m3front. The vast bulk of the change is in CG.m3, with a small change in Variable.m3 to pass it the bounds. It took me a while to cope with the mixture of signed and unsigned that the front end throws at its CG.m3. More can be done here. e.g. mod a positive non-zero constant returns 0..n-1. min and max(a,b) has bounds computable from the bounds of a and b. abs returns a positive number, except for the overflow case neg(abs) returns 0 or negative (again with some chance of overflow) I think the change is ok. There is the small matter of how to call GetBounds. I find it a little wierd that some versions of GetBounds return a boolean, some do not. The signed/unsigned cases could be combined down somehow, replacing min/max in some places with zero. Load_addr_of_temp should probably use WITH. Other forms of Load e.g. Load_indirect should probably be changed analogously. There is also a matter of "bounds" for non-ordinal types. For example a set might be a constant. It might be possible to reason about floating point. But I didn't do any this stuf. Just ordinal types. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 05:15:22 +0000 > Prep vs. PrepBR vs. Compile Expr.i3 documents it. (*** phase 4 ****) (* Expressions are compiled in two steps: Prep: emit any code that includes branchs or stores Compile: emit the rest of the code *) PROCEDURE Prep (t: T); PROCEDURE Compile (t: T); (* emits code to evaluate the expression onto the top of stack *) PROCEDURE PrepLValue (t: T; traced: BOOLEAN); PROCEDURE CompileLValue (t: T; traced: BOOLEAN); (* emits code to evaluate 't's L-value into s0.A. *) PROCEDURE CompileAddress (t: T; traced: BOOLEAN); (* emits code to evaluate 't's byte address onto the top of stack. Use PrepLValue to prep these expressions. *) PROCEDURE PrepBranch (t: T; true, false: CG.Label; freq: CG.Frequency); PROCEDURE CompileBranch (t: T; true, false: CG.Label; freq: CG.Frequency); (* emits code to evaluate the expression and conditionally branch to 'true' or 'false' depending on its boolean value. 'freq' is the estimated frequency that the specified branch will be taken. *) - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 05:01:24 +0000 Hm. How about elsewhere m3front? Optimizing middle end not necessarily a different package? (esp. given that m3front is already big enough to afford being organized into directories: m3front/src/middle, m3front/src/target-independent-optimizations?) Maybe elsewhere in the two files I changed? Prep or PrepBR or Compile instead of Fold? I don't understand this Prep vs. PrepBR business. I found out that "BR" stands for branch, but that doesn't explain it me. Also PrepBR and Compile look very similar. I also noticed that Fold gets called twice (I had some RTIO in it) but didn't look into why. Surely doing it in PrepBR or Compile is correct or CG.m3 is correct, reasonable, efficient, maintainable? I can see why Fold would be wrong -- allowing code to compile, with a reasonable meaning, but that isn't supposed to. Or in CG.m3, which is conceptually "the bottom" of m3front, any two adjacent pieces can be considered merged into one layer. I'll look at both of those options later. I think "subrange analysis" if done enough might yield efficiencies in real code. Though m3front might already do enough of it -- e.g. the fact that you can't modify a FOR loop index affords some efficiencies, that are probably already taken into account. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] comparisons vs. subranges Date: Sun, 14 Mar 2010 03:52:15 +0000 Hm. You mean, like I'm not supposed to be able to say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (a < b) => IO.Put("true\n"); | (a > b) => IO.Put("false\n"); END; but I can say: PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= BEGIN CASE bool OF | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); END; (I'd like to be able to say FIRST(a), etc., but I don't think it is allowed.) "constants" kind of seem like an optimization themselves. Other than efficiency concerns, they could all be variables initialized by module initializers, that the frontend doesn't let you write to. Including de-optimizations like runtime allocation/sizing of arrays based on const/non-const sizes. Look at how I changed const to var already in m3core/unix, and then had to change CASE to if/elseif ladders. It's a minor matter of what the language lets you say, among various basically equivalent forms. The compiler could allow case to target non-constants. It'd just mainly be slower. There is the matter of CASE arms can't overlap, where if/elseif ladders can. Because CASE is conceptually compared all at once where if/elseif is sequential. Ok. I see a few options at least. m3middle could look basically like m3front, but with various checks like type checking removed and assumed true. This would be an arduous task of duplication. M3CG.i3 Var and Target.Int could gain the following fields (or methods): min_valid := FALSE; max_valid := FALSE; min := TInt.Zero; max := TInt.Zero; m3front could fill them in a few cases. Initially none, but then a few as they are discovered to be easy, correct, efficient. For constants, min = max value. For subranges, min/max come the type. The backends could use them (if valid) and possibly transform them. e.g. MIN(a,b) with valid min/max yields an expression where min is the min of the two mins, max is the min of the two maxes. MIN([0..1],[2..3]) < 2 => TRUE. Probably more correct would be: PROCEDURE declare_enum (xx: T; t: TypeUID; n_elts: INTEGER; s: BitSize) = PROCEDURE declare_subrange (xx: T; t, domain: TypeUID; READONLY min, max: Target.Int; s: BitSize) => already exist and then PROCEDURE load (xx: T; v: Var; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_indirect (xx: T; o: ByteOffset; t: MType; u: ZType) = PROCEDURE load_integer (xx: T; t: IType; READONLY i: Target.Int) = PROCEDURE load_ordered (xx: T; t: MType; u: ZType; order: MemoryOrder) = etc. should all take an optional TypeUID. The second proposal I've thought a bit more about and it seems very easy. The last idea seems cleaner, but with possibly signifiant downside in that e.g. Word.T is not a subrange. There's also some unclarity with respect to Word.LT(expression, -1); -1 is a valid Word interpreted as a large number. One would want to be careful not to break that. I believe this is related to what you were saying, where operations are typed, not values. It could be that every operation needs min/max or typeuid (pairs of them for binary operations). Can we take a stab at something like this? This might also serve to replace some of what m3back does with constants already, since constants would fall out of subranges of length 1. Thanks, -Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 14:45:29 -0500 To: hosking at cs.purdue.edu CC: m3devel at elegosoft.com; jay.krell at cornell.edu Subject: Re: [M3devel] comparisons vs. subranges Jay, I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 14:23, Tony Hosking wrote: On 13 Mar 2010, at 13:26, Jay K wrote: Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. What is the point of these functions Fold? So that constants can be manipulated as first-class values in the language. See http://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. Why does it have these various "cost" computations? Which cost computations are you referring to? Granted, they are rough. Should we beef up m3middle? That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. Designing a decent optimising middle-end takes significant time and effort, and will require more thought. m3back is correct or extremely close to correct as far as I know. It is missing atomic operations for 8, 16, 64 bit operands. (PPC32 seems unable to support 64bit atomics btw.) Correct. I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. - Jay From: hosking at cs.purdue.edu Date: Sat, 13 Mar 2010 13:03:18 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] comparisons vs. subranges Jay, I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 13 Mar 2010, at 05:19, Jay K wrote: <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; Seems to me, the front end should notice these. The first should always be TRUE. And possibly, possibly warn. The second should always be FALSE. And possibly, possibly warn. "Generic" programming often hits this sort of thing though, a good reason not to warn. Programmer might also be working with existing code and have changed INTEGER to CARDINAL. Or be defending against future maintainers changing CARDINAL to INTEGER. The backend isn't give enough information, because CARDINAL = INTEGER as far as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. The only way to get unsigned types is to use ADDRESS from what I see. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 1.txt URL: From rodney_bates at lcwb.coop Sun Mar 14 16:20:29 2010 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 14 Mar 2010 10:20:29 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: <20100313182924.GA14329@topoi.pooq.com> References: <20100313182924.GA14329@topoi.pooq.com> Message-ID: <4B9CFEBD.8050309@lcwb.coop> hendrik at topoi.pooq.com wrote: > > Wasn't there a discussion a while ago about subranges out-of-bounds not > being a safety problem? (Or was it arithmetic overflow?) This > optimisation might well cause a quite hard-to-find bug if we don't > guarantee subrange integrity. Subrange is a safety problem. Overflow is not, although it can be a valuable way for the language/implementation to help find bugs. Most of the definitions of safety are neither very useful nor consistent with common, informal usage. My definition is that safety means everything that can happen can be explained and understood using only the concepts of the programming language. You don't have to resort to knowing about machine level stuff, especially bit representations and the fact that things that are entirely autonomous in the language (e.g. separately declared variables) actually occupy the same homogeneous memory, along with machine code and all sorts of other stuff. So, if LAST(INTEGER)+LAST(INTEGER) overflows, even if it is undefined by the language what happens, all the possibilities are still comprehensible in terms of the language. Either you get some value that is a member of INTEGER, or an exception, all of which are language concepts. But if you could assign 16_FF to a variable of type [0..20], you get something that can only be understood by looking at the machine-level representation. Most likely, it is a bit pattern that would represent a value that is not a member of the variable's type. This is a safely problem. BTW, it's also implementation- and target-dependent. > > -- hendrik > From rodney_bates at lcwb.coop Sun Mar 14 16:58:21 2010 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 14 Mar 2010 10:58:21 -0500 Subject: [M3devel] bandwith In-Reply-To: References: Message-ID: <4B9D079D.1060207@lcwb.coop> Dirk Muysers wrote: > > To make it work for all these different platforms seems a herculean task > and rather pointless > endeavour. Why not use all that energy to try a radically disruptive > strategy such as a platform > agnostic intermediate form targeting a JIT. > I have been a little way down this road at least three times, with at least two different languages, and it turns out not to be what it would appear at first glance, at least for the JVM. Java is a mostly object-oriented language, meaning the non-object-oriented types and operations are very limited. Most of the interesting stuff can be done only with full-blown heap allocated objects and dispatching methods. Even the equivalents of records and arrays are this way inb Java. When you don't need this full generality and power, you pay big performance penalties for heap allocation, and this is worse and much deeper than the performance penalties of an interpretive implementation. (which, of course, a JIT code generator partially avoids, and a traditional source-to-machine compiler avoids altogether, all with no language changes.) Many languages, Modula-3 included, give this generality for when you need it, but also give you lighter weight alternatives, for use when appropriate. So when you try to implement some other language using JVM, you soon discover that JVM lacks a lot of operations needed for the not-so- impoverished non-object-oriented types of the language. Whether or how badly the .net/mono virtual machine suffers from this, I don't know. It apparently was designed to support a much wider range of languages, but I seem to recall hearing some discussions that suggested it had some of the same limitations. Of course, we could invent our own virtual machine (M-code? MVM?) ;-). This whole approach is of course, not very suitable for systems programming, one of Modula-3's strengths. > d. muysers From rodney_bates at lcwb.coop Sun Mar 14 17:54:58 2010 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 14 Mar 2010 11:54:58 -0500 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: <4B9D14E2.6020403@lcwb.coop> Jay K wrote: > Hm. You mean, like I'm not supposed to be able to say: > > > PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= > BEGIN > CASE bool OF > | (a < b) => IO.Put("true\n"); > | (a > b) => IO.Put("false\n"); > END; > No, because a and b are not constants, and thus neither is a < b. Use an IF ladder for this. > > but I can say: > > > PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= > BEGIN > CASE bool OF > | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); > | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); > END; Yes. You can apply FIRST to a(non-open) array or ordinal _type_. > > (I'd like to be able to say FIRST(a), etc., but I don't think it is > allowed.) > Yes, or rather no. Yes, it's not allowed. Yes, we have no bananas. You can apply FIRST to a _variable_ of array type and it will be legal and (if not an open array), constant. You can't apply FIRST to a variable of ordinal type. While probably not very useful, it has always seemed to me to be a bit inconsistent that you can't. It might be useful in some kind of generated, generic, or otherwise parameterized code. If you intend these examples just to talk about language rules, OK. If examples of how to write actual code, they seem convoluted to me. They are better candidates for IF ladders. When the type of the expression is BOOLEAN, probably a single IF statement, with ELSE, is in order. The only situation I can think of where I would code something like this is if I wanted to force a compile-time error if the two expressions in the two alternatives ever became equal during maintenance, because someone changed one of the constants or types in a way that violated some algorithmic constraint. Or maybe just to emphasize that the case label expression should be constant and get the compiler to verify that. From hosking at cs.purdue.edu Sun Mar 14 19:28:56 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 14 Mar 2010 14:28:56 -0400 Subject: [M3devel] comparisons vs. subranges In-Reply-To: References: , , , , , <2AFF4165-F3E3-42D8-A266-8FF510DAACFA@cs.purdue.edu> Message-ID: Correct. On 13 Mar 2010, at 22:52, Jay K wrote: > Hm. You mean, like I'm not supposed to be able to say: > > > PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= > BEGIN > CASE bool OF > | (a < b) => IO.Put("true\n"); > | (a > b) => IO.Put("false\n"); > END; > > > but I can say: > > > PROCEDURE F1(bool:BOOLEAN;a:[0..1];b:[2..3])= > BEGIN > CASE bool OF > | (LAST([0..1]) < FIRST([2..3]) => IO.Put("true\n"); > | (FIRST([0..1]) > LAST([2..3])=> IO.Put("false\n"); > END; > > (I'd like to be able to say FIRST(a), etc., but I don't think it is allowed.) > > > > "constants" kind of seem like an optimization themselves. > Other than efficiency concerns, they could all be variables > initialized by module initializers, that the frontend doesn't > let you write to. Including de-optimizations like runtime > allocation/sizing of arrays based on const/non-const sizes. > > > Look at how I changed const to var already in m3core/unix, and then > had to change CASE to if/elseif ladders. > It's a minor matter of what the language lets you say, > among various basically equivalent forms. > The compiler could allow case to target non-constants. > It'd just mainly be slower. > There is the matter of CASE arms can't overlap, where if/elseif ladders can. > Because CASE is conceptually compared all at once where if/elseif > is sequential. > > > Ok. > > > > I see a few options at least. > > > m3middle could look basically like m3front, but with various checks like type checking removed > and assumed true. This would be an arduous task of duplication. > > M3CG.i3 Var and Target.Int could gain the following fields (or methods): > > > min_valid := FALSE; > max_valid := FALSE; > min := TInt.Zero; > max := TInt.Zero; > > > m3front could fill them in a few cases. > Initially none, but then a few as they are discovered to be easy, correct, efficient. > For constants, min = max value. > For subranges, min/max come the type. > The backends could use them (if valid) and possibly transform them. > e.g. MIN(a,b) with valid min/max yields an expression > where min is the min of the two mins, max is the min of the two maxes. > > > MIN([0..1],[2..3]) < 2 => TRUE. > > > Probably more correct would be: > > PROCEDURE declare_enum (xx: T; t: TypeUID; n_elts: INTEGER; s: BitSize) = > > PROCEDURE declare_subrange (xx: T; t, domain: TypeUID; > READONLY min, max: Target.Int; > s: BitSize) > > => already exist > > > and then > > > PROCEDURE load (xx: T; v: Var; o: ByteOffset; t: MType; u: ZType) = > PROCEDURE load_indirect (xx: T; o: ByteOffset; t: MType; u: ZType) = > PROCEDURE load_integer (xx: T; t: IType; READONLY i: Target.Int) = > PROCEDURE load_ordered (xx: T; t: MType; u: ZType; order: MemoryOrder) = > etc. > > > should all take an optional TypeUID. > > > The second proposal I've thought a bit more about and it seems very easy. > The last idea seems cleaner, but with possibly signifiant downside > in that e.g. Word.T is not a subrange. > > There's also some unclarity with respect to > Word.LT(expression, -1); > > -1 is a valid Word interpreted as a large number. > One would want to be careful not to break that. > I believe this is related to what you were saying, where operations > are typed, not values. > > > It could be that every operation needs min/max or typeuid (pairs > of them for binary operations). > > > Can we take a stab at something like this? > > > This might also serve to replace some of what m3back does with constants already, > since constants would fall out of subranges of length 1. > > > Thanks, > -Jay > > > From: hosking at cs.purdue.edu > Date: Sat, 13 Mar 2010 14:45:29 -0500 > To: hosking at cs.purdue.edu > CC: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: Re: [M3devel] comparisons vs. subranges > > Jay, > > I have reverted your commits because they violate the language definition. Constant folding in the front-end is there only to support ha language definition of constant expressions. Your changes violated that spec. If you want such optimisations they should be performed in the back-end, or if we ever get one, in an optimising middle-end. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 13 Mar 2010, at 14:23, Tony Hosking wrote: > > On 13 Mar 2010, at 13:26, Jay K wrote: > > Tony these are simple target-independent optimizations. I would dislike for each backend to do target-independent optimizations. Where can we put those? The front end currently does several. For example, operations on sets that fit in an integer. Division by powers of 2, Unrolling comparisons of records and sets (at least up to a certain size), removing runtime operations that are known to violate subranges (e.g. in insert/extract), occasional constant folding (in the same functions I modified), etc. > > > What is the point of these functions Fold? > > So that constants can be manipulated as first-class values in the language. Seehttp://www.cs.purdue.edu/homes/hosking/m3/reference/constexpr.html. There are many places where a compile-time constant expression is required. Fold is *not* intended for performing optimisations. > > Why does it have these various "cost" computations? > > Which cost computations are you referring to? > > Granted, they are rough. > > > Should we beef up m3middle? > > That might be the place... but ad hoc add-ons to m3front like you have been leaning towards are probably not the right place. Designing a reasonable place for all of these optimisations will take effort. I don't want to see us degrading the maintainability of m3front with gratuitous optimisations of dubious value. > > m3back doesn't optimize comparisons to constants, and a *lot* of what it does is target-independent. > It would be nice to factor it better so that x86-independent code is not in files with "x86" in their name. > It'd be nice to extend it to AMD64 for example (which is why I was worrying about my notions of "TypeIs64" and multiplying register counts times 4 to get byte counts, etc..) and maybe even to Sparc, PPC, ARM, etc. > > Designing a decent optimising middle-end takes significant time and effort, and will require more thought. > > m3back is correct or extremely close to correct as far as I know. > It is missing atomic operations for 8, 16, 64 bit operands. > (PPC32 seems unable to support 64bit atomics btw.) > > Correct. > > I have to test the subrange stuff for 64bit operands. It might work, but I think optimizations are missing there as well. > > > I'd also like to maybe inline more operations..and I was thinking about implementing inlining in general. It might not be so difficult. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Sat, 13 Mar 2010 13:03:18 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] comparisons vs. subranges > > Jay, > > I am disturbed that you are inserting optimisations into the front-end that don't really belong there. The front-end is responsible for semantics and not optimisation. It was never designed to be an optimising compiler. There are better ways to go about working in optimisation, but I would hate to muddy the waters of the front-end the way you seem to intend. Let's not go down this path until there is consensus! One approach would involve communicating more information to the "back"-ends (actually, the gcc back-end should be considered a middle-end from the global perspective of optimising compilers). For example, the back-ends gets very little in the way of type information (as you note w.r. to CARDINAL). In any case, I suggest that you not obsess about performance right now but simply concentrate on correctness in your backend. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 13 Mar 2010, at 05:19, Jay K wrote: > > <*UNUSED*>PROCEDURE CardinalGE0(a:CARDINAL):BOOLEAN=BEGIN RETURN a>=0; END CardinalGE0; > <*UNUSED*>PROCEDURE CardinalEQN1(a:CARDINAL):BOOLEAN=BEGIN RETURN a=-1; END CardinalEQN1; > > > Seems to me, the front end should notice these. > The first should always be TRUE. > And possibly, possibly warn. > The second should always be FALSE. > And possibly, possibly warn. > > "Generic" programming often hits this sort of thing though, a good reason not to warn. > Programmer might also be working with existing code and have changed INTEGER to CARDINAL. > Or be defending against future maintainers changing CARDINAL to INTEGER. > > > The backend isn't give enough information, because CARDINAL = INTEGER as far > as it is told. Due to the half range of CARDINAL and LONGCARD, that isn't wrong. > The only way to get unsigned types is to use ADDRESS from what I see. > > > - Jay > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 14 19:30:59 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 14 Mar 2010 18:30:59 +0000 Subject: [M3devel] atomics vs. locks.. Message-ID: http://msdn.microsoft.com/en-us/library/ee418650(VS.85).aspx MemoryBarrier was measured as taking 20-90 cycles. InterlockedIncrement was measured as taking 36-90 cycles. Acquiring or releasing a critical section was measured as taking 40-100 cycles. Acquiring or releasing a mutex was measured as taking about 750-2500 cycles perhaps just use critical sections... (I'm very ambivalent about atomics. It is just so darn difficult to do lockless programming. Granted, if you are a systems language, then you need to be able to imlement the locks themselves. Perhaps.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Sun Mar 14 22:20:26 2010 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sun, 14 Mar 2010 21:20:26 +0000 (GMT) Subject: [M3devel] bandwith Message-ID: <646699.86796.qm@web23603.mail.ird.yahoo.com> Hi Rodney: Well, I think the main issue as you point out is the architecture of the VM, as you know JVM is a Java based architecture so many of its structures are suited for that language and probably this is related to a compiler efficiency thing and thus you can say that a big language like Java needs an appropiate VM like JVM. I think that in M3 case what could be more suitable for real use is a SPIN powered vm like itself running in bare hardware with current hardware assisted virtualization such of current leading processors that would be a plus, it would lack supoort for more real machine languages (i.e C), but as it is running in kernel mode could give a better performance. In other architectures it could be run with kernel level virtualization and thus gaining better performance or similar even if its not a complete form of virtualization in bare hardware (by writing for SPIN M3 or other VM based language too and run in kernel mode like a M3 process, like the JVM M3 hosted if so, etc). I think having a kernel level virtual machine could be a gain in performance for some applications written for other machines or languages and or virtual machines or even the same kind of machines in M3 but in other systems (like running a DEC Unix or BSD flawor even with C libraries from inside SPIN). In fact SPIN was ported to NT as a driver for an intelligent NIC, thus allowed to run in bare hardware and at kernel level Thanks in advance, --- El dom, 14/3/10, Rodney M. Bates escribi?: > De: Rodney M. Bates > Asunto: Re: [M3devel] bandwith > Para: m3devel at elegosoft.com > Fecha: domingo, 14 de marzo, 2010 10:58 > > > Dirk Muysers wrote: > >? To make it work for all these different > platforms seems a herculean task and rather pointless > > endeavour. Why not use all that energy to try a > radically disruptive strategy such as a platform > > agnostic intermediate form targeting a JIT. > >? > > I have been a little way down this road at least three > times, with at > least two different languages, and it turns out not to be > what it would > appear at first glance, at least for the JVM. > > Java is a mostly object-oriented language, meaning the > non-object-oriented > types and operations are very limited.? Most of the > interesting stuff can > be done only with full-blown heap allocated objects and > dispatching methods. > Even the equivalents of records and arrays are this way inb > Java.? When you > don't need this full generality and power, you pay big > performance penalties > for heap allocation, and this is worse and much deeper than > the > performance penalties of an interpretive implementation. > (which, of > course, a JIT code generator partially avoids, and a > traditional > source-to-machine compiler avoids altogether, all with no > language > changes.) > > Many languages, Modula-3 included, give this generality for > when you need > it, but also give you lighter weight alternatives, for use > when appropriate. > > So when you try to implement some other language using JVM, > you soon > discover that JVM lacks a lot of operations needed for the > not-so- > impoverished non-object-oriented types of the language. > > Whether or how badly the .net/mono virtual machine suffers > from this, I > don't know.? It apparently was designed to support a > much wider range > of languages, but I seem to recall hearing some discussions > that suggested > it had some of the same limitations. > > Of course, we could invent our own virtual machine (M-code? > MVM?) ;-). > > This whole approach is of course, not very suitable for > systems programming, > one of Modula-3's strengths. > > > > > d. muysers > From jay.krell at cornell.edu Mon Mar 15 03:51:54 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 15 Mar 2010 02:51:54 +0000 Subject: [M3devel] Mx86.m3 vs. stackx86.m3? Message-ID: Does anyone understand and can explain the rationale to put some things in Mx86.m3 and some in Stackx86.m3? It seems arbitrary in many places to me. For exampe, in the release branch, compare M3x86 shift, shift_left, shift_right. shift_left and shift_right do the work themselves, but shift calls stack.doshift. In this I case merged all the code into stack.doshift, with an enum to indicate which of the three types of shifts to do. rotate is still the old way. (I don't yet inline 64bit rotates without constants.) Also, I assume "vstack" means "virtual stack"? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 16 01:54:41 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 00:54:41 +0000 Subject: [M3devel] arbitrary use of LONGINT for testing? target/ABI alteration/generation via command line? Message-ID: Tony et. al. I'm wondering/fishing if you any ideas, like a cm3 command line switch, that might be used to make INTEGER 64bits to increase coverage/testing of 64bit integer support. The problem is *at least* interfacing with external code/files, if not breaking everything. Maybe also a problem around sizeof(INTEGER) vs. sizeof(ADDRESS). Probably such a setting would grow ADDRESS too. Like, maybe stuff like m3core/unix, m3core/win32 should never use plain INTEGER or LONGINT, but "BITS FOR", to allow this to work? Or maybe <* EXTERNAL *> would be the hint? No, <* EXTERNAL *> isn't adequate, due to the case of callbacks. I'd like to somehow, without too much effort, significantly increase testing of LONGINT. Maybe a wierdo platform NT386_64 that does this just for test purposes? I might try that, if "BITS FOR" is enough to actually let it work (interface with C). (or generally, cm3 -test64 would append "_TEST64" to target name, so we could have LINUXLIBC6_TEST64, I386_DARWIN_TEST64, etc.) I can do some initial experiments along this -test64 line, see if it completely blows up or not. A few minutes thought suggests it should work easily and provide value. (There's probably something similar to try around a 16 bit CHAR. vs. BITS 8 FOR CHAR, but I'm not going there..) Such command line driven target/ABI alterations seem like a somewhat good idea. e.g. cm3 -userthreads would append _USERTHREADS. cm3 -extended80 would use an 80 or 96 bit extended on x86 (96 bits for alignment). -extended128 on ppc etc. cm3 -msvc80 could probe around in enviroment for a Visual C++ 8.0 compiler/linker/header/libraries and append "_MSVC80" to target, else fail. I'll just try -test64 for now. The rest aren't as immediately useful. -userthreads would be my next "favorate". Hm. Instead of -test64, how about -64. If integer is 32 bits, change it to 64 and append _64 (and address). If integer is already 64bits, do nothing. This is quite different than gcc's -m64 though. Maybe not good that way. You can see parallels in several older C compilers. Metrowerks for Mac/68K let you chose integer size and either double or extended size (I forget which). That gave you a cross product set of ABIs, each with their own libraries. Similarly you could chose to issue FPU instructions directly, which might have altered the ABI, or maybe the alternate libraries were just faster. Similarly Microsoft and memory models. Various switches changed the size of a pointer and implied a need for different libraries. Current gcc and ppc floating point sizes I believe is a contemporary example. Even sometimes the affect that -pthread has -- chosing different libraries (unnecessary mess imho, but one that seems to persist on HP-UX.) It's kind of an ugly area, to offer too many choices, produces confusion, but the limited important goal of increasing 64bit integer testing seems worth going here. And increasing userthread testing. People have expressed an interest in higher precision floating point, esp. 80 bit on x86, so this maybe a good way to proceed there also. You could also imagine flags -SSE, -SSE2, etc. It is important to only support a very limited number of these flags. Large cross products are not fun to worry about and test. But for example, with SSE, that nets you several targets all at once, without per-target work. (I lose the term "SSE" loosely. Could be I mean "SSE2" or 3 -- whatever is sufficient to largely replace the stack based x87.) - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 02:00:27 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 15 Mar 2010 20:00:27 -0500 Subject: [M3devel] arbitrary use of LONGINT for testing? target/ABI alteration/generation via command line? In-Reply-To: References: Message-ID: <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu> Let's not right now. We need to get the release out. I suggest a freeze on compiler changes for now. Bug fixes only! Sent from my iPhone On Mar 15, 2010, at 7:54 PM, Jay K wrote: > Tony et. al. I'm wondering/fishing if you any ideas, like a cm3 > command line switch, that might be used to make INTEGER 64bits to > increase coverage/testing of 64bit integer support. > > > The problem is *at least* interfacing with external code/files, if > not breaking everything. > Maybe also a problem around sizeof(INTEGER) vs. sizeof(ADDRESS). > Probably such a setting would grow ADDRESS too. > > > Like, maybe stuff like m3core/unix, m3core/win32 should never use > plain INTEGER or LONGINT, but "BITS FOR", to allow this to work? Or > maybe <* EXTERNAL *> would be the hint? > No, <* EXTERNAL *> isn't adequate, due to the case of callbacks. > > > I'd like to somehow, without too much effort, significantly increase > testing of LONGINT. > > > Maybe a wierdo platform NT386_64 that does this just for test > purposes? > I might try that, if "BITS FOR" is enough to actually let it work > (interface with C). > (or generally, cm3 -test64 would append "_TEST64" to target name, > so we could have LINUXLIBC6_TEST64, I386_DARWIN_TEST64, etc.) > > > I can do some initial experiments along this -test64 line, see if it > completely blows up or not. > A few minutes thought suggests it should work easily and provide > value. > > > (There's probably something similar to try around a 16 bit CHAR. vs. > BITS 8 FOR CHAR, but I'm not going there..) > > > Such command line driven target/ABI alterations seem like a somewhat > good idea. > e.g. cm3 -userthreads would append _USERTHREADS. > cm3 -extended80 would use an 80 or 96 bit extended on x86 (96 bits > for alignment). > -extended128 on ppc > etc. > > > cm3 -msvc80 could probe around in enviroment for a Visual C++ 8.0 > compiler/linker/header/libraries and append "_MSVC80" to target, > else fail. > > > I'll just try -test64 for now. > The rest aren't as immediately useful. > -userthreads would be my next "favorate". > > Hm. Instead of -test64, how about -64. > If integer is 32 bits, change it to 64 and append _64 (and address). > If integer is already 64bits, do nothing. > > > This is quite different than gcc's -m64 though. > Maybe not good that way. > > > You can see parallels in several older C compilers. > > Metrowerks for Mac/68K let you chose integer size and either double > or extended size (I forget which). > That gave you a cross product set of ABIs, each with their own > libraries. > > > Similarly you could chose to issue FPU instructions directly, which > might have altered the ABI, or maybe the alternate libraries were > just faster. > > > Similarly Microsoft and memory models. Various switches changed the > size of a pointer and implied a need for different libraries. > > > Current gcc and ppc floating point sizes I believe is a contemporary > example. > Even sometimes the affect that -pthread has -- chosing different > libraries (unnecessary > mess imho, but one that seems to persist on HP-UX.) > > > It's kind of an ugly area, to offer too many choices, produces > confusion, but the limited important goal of increasing 64bit > integer testing seems worth going here. And increasing userthread > testing. People have expressed an interest in higher precision > floating point, esp. 80 bit on x86, so this maybe a good way to > proceed there also. > You could also imagine flags -SSE, -SSE2, etc. > > > It is important to only support a very limited number of these flags. > Large cross products are not fun to worry about and test. > But for example, with SSE, that nets you several targets all at > once, without per-target work. > > (I lose the term "SSE" loosely. Could be I mean "SSE2" or 3 -- > whatever is sufficient to largely replace the stack based x87.) > > > - Jay From jay.krell at cornell.edu Tue Mar 16 02:41:37 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 01:41:37 +0000 Subject: [M3devel] release status [was something else] In-Reply-To: <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu> References: , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu> Message-ID: Release status as far as I know: - cvsupd hang Reported over a year ago. Needs investigation. Can anyone confirm it with a recent version? And, as Tony asked, with user threads? - NT386 has no 64bit longint in release branch. Partly the point of this question. - Should maybe improve build/packaging to get NT386-VC80.msi, NT386-VC90.msi etc. (not the same as "target/abi alteration via command line" expressed below, though that is a possibility, but separate clean builds for now and/or separate CVS checkouts); really more of a Hudson task now, to have multiple build environments installed and used, I think I already updated the .msi building to append the tag. Olaf? - I believe there is "new" divergence in m3front between head and release -- whatever came along with the TInt changes. Otherwise I'd have to look through trac. There's maybe some stuff not merged from release to head, but that's ok for release. (I need to check which branch Randy fixed the examples in.) I need to re-test the .msis. - Jay > From: hosking at cs.purdue.edu > To: jay.krell at cornell.edu > Subject: Re: [M3devel] arbitrary use of LONGINT for testing? target/ABI alteration/generation via command line? > Date: Mon, 15 Mar 2010 20:00:27 -0500 > CC: m3devel at elegosoft.com > > Let's not right now. We need to get the release out. I suggest a > freeze on compiler changes for now. Bug fixes only! > > Sent from my iPhone > > On Mar 15, 2010, at 7:54 PM, Jay K wrote: > > > Tony et. al. I'm wondering/fishing if you any ideas, like a cm3 > > command line switch, that might be used to make INTEGER 64bits to > > increase coverage/testing of 64bit integer support. > > > > > > The problem is *at least* interfacing with external code/files, if > > not breaking everything. > > Maybe also a problem around sizeof(INTEGER) vs. sizeof(ADDRESS). > > Probably such a setting would grow ADDRESS too. > > > > > > Like, maybe stuff like m3core/unix, m3core/win32 should never use > > plain INTEGER or LONGINT, but "BITS FOR", to allow this to work? Or > > maybe <* EXTERNAL *> would be the hint? > > No, <* EXTERNAL *> isn't adequate, due to the case of callbacks. > > > > > > I'd like to somehow, without too much effort, significantly increase > > testing of LONGINT. > > > > > > Maybe a wierdo platform NT386_64 that does this just for test > > purposes? > > I might try that, if "BITS FOR" is enough to actually let it work > > (interface with C). > > (or generally, cm3 -test64 would append "_TEST64" to target name, > > so we could have LINUXLIBC6_TEST64, I386_DARWIN_TEST64, etc.) > > > > > > I can do some initial experiments along this -test64 line, see if it > > completely blows up or not. > > A few minutes thought suggests it should work easily and provide > > value. > > > > > > (There's probably something similar to try around a 16 bit CHAR. vs. > > BITS 8 FOR CHAR, but I'm not going there..) > > > > > > Such command line driven target/ABI alterations seem like a somewhat > > good idea. > > e.g. cm3 -userthreads would append _USERTHREADS. > > cm3 -extended80 would use an 80 or 96 bit extended on x86 (96 bits > > for alignment). > > -extended128 on ppc > > etc. > > > > > > cm3 -msvc80 could probe around in enviroment for a Visual C++ 8.0 > > compiler/linker/header/libraries and append "_MSVC80" to target, > > else fail. > > > > > > I'll just try -test64 for now. > > The rest aren't as immediately useful. > > -userthreads would be my next "favorate". > > > > Hm. Instead of -test64, how about -64. > > If integer is 32 bits, change it to 64 and append _64 (and address). > > If integer is already 64bits, do nothing. > > > > > > This is quite different than gcc's -m64 though. > > Maybe not good that way. > > > > > > You can see parallels in several older C compilers. > > > > Metrowerks for Mac/68K let you chose integer size and either double > > or extended size (I forget which). > > That gave you a cross product set of ABIs, each with their own > > libraries. > > > > > > Similarly you could chose to issue FPU instructions directly, which > > might have altered the ABI, or maybe the alternate libraries were > > just faster. > > > > > > Similarly Microsoft and memory models. Various switches changed the > > size of a pointer and implied a need for different libraries. > > > > > > Current gcc and ppc floating point sizes I believe is a contemporary > > example. > > Even sometimes the affect that -pthread has -- chosing different > > libraries (unnecessary > > mess imho, but one that seems to persist on HP-UX.) > > > > > > It's kind of an ugly area, to offer too many choices, produces > > confusion, but the limited important goal of increasing 64bit > > integer testing seems worth going here. And increasing userthread > > testing. People have expressed an interest in higher precision > > floating point, esp. 80 bit on x86, so this maybe a good way to > > proceed there also. > > You could also imagine flags -SSE, -SSE2, etc. > > > > > > It is important to only support a very limited number of these flags. > > Large cross products are not fun to worry about and test. > > But for example, with SSE, that nets you several targets all at > > once, without per-target work. > > > > (I lose the term "SSE" loosely. Could be I mean "SSE2" or 3 -- > > whatever is sufficient to largely replace the stack based x87.) > > > > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 16 11:39:09 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 11:39:09 +0100 Subject: [M3devel] release status [was something else] In-Reply-To: References: , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu> Message-ID: <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Quoting Jay K : > Release status as far as I know: > > - cvsupd hang > Reported over a year ago. Needs investigation. > Can anyone confirm it with a recent version? > And, as Tony asked, with user threads? If we can reproduce this, we should fix it; if not, I'd say this isn't relevant for the release. We can try to use a cvsupd from the release branch on birch serving the cm3 repository on a different port. Let's see if that works. Maybe tonight. > - NT386 has no 64bit longint in release branch. Partly the point of > this question. I thought Randy was going to do some more testing. If the m3back LONGINT changes don't break anything else, I'd be for just merging them to the branch and see if the builds and tests succeed. Of course testing with some other real applications would be great, but we must not delay endlessly for that. > - Should maybe improve build/packaging to get NT386-VC80.msi, > NT386-VC90.msi etc. (not the same as "target/abi alteration via > command line" expressed below, though that is a possibility, but > separate clean builds for now and/or separate CVS checkouts); really > more of a Hudson task now, to have multiple build environments > installed and used, I think I already updated the .msi building to > append the tag. Olaf? I'd rather not touch the NT386 setup on our virtual machine; perhaps you could provide another machine to run the Hudson jobs with a different Microsoft tools setup? > - I believe there is "new" divergence in m3front between head and > release -- whatever came along with the TInt changes. Is this relevant for the release? Or can we just ignore it? > Otherwise I'd have to look through trac. That's always a good idea. > There's maybe some stuff not merged from release to head, but that's > ok for release. > > (I need to check which branch Randy fixed the examples in.) > > I need to re-test the .msis. We still need to add them to the WWW export/display, don't we? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 16 12:21:36 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 11:21:36 +0000 Subject: [M3devel] release status [was something else] In-Reply-To: <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: Responding to just parts: > > - NT386 has no 64bit longint in release branch. Partly the point of > > this question. > > I thought Randy was going to do some more testing. If the m3back LONGINT > changes don't break anything else, I'd be for just merging them to the It is a big change. It is dependent on TInt/TWord changes, though that is flexible. I could copy and rename them "M3BackInt". Or we could take them -- they go with m3front changes. Or we could even do a little experiment: The reason I use TInt/TWord is to get 64bit integers portably to 32bit hosts. Previously INTEGER/Word was used. I could try using LONGINT/Long instead. However it is still a large change. There are some other unrelated changes, nothing too significant on its own (I always say), but it adds up: It isn't messed up, but it is maybe "beyond recognition" compared to the previous. (as was recently suggested) some renaming for, I claim, clarity some small optimizations (e.g. using shorter encodings sometimes, like a byte instead of a dword for constants in instructions, or equivalent shorter constructs like or reg,-1 instead of mov reg,-1), atomics support, actually in very good shape now, but could use more testing and optimization "rewrite" of insert/extract to no longer use the tables, I've always been bothered by those tables "rewrite" of set_singleton/set_member to be *much* smaller/faster set_singleton is just the bts instruction, instead of a function call set_member is just bt instruction plus carry flag extraction, instead of a function call cleanup of the way epilogues are sometimes "patched in" Only save/restore non-volatile registers that are actually used. Probably other stuff I'm forgetting. There's also small related changes in m3objfile: support for outputing more than 4 bytes at a time, support for "backing up". Some extra commenting. Some formating consistency (then always at end of line, else always on its own line). "Namespace flattening" (from foo import a,b,c). I need to test 64bit subrange checking. The code is a little fishy in that it deals with single registers. It is an optimization and could probably just be pessimized. Or maybe easy to fix. There's also the question as to if the TInt use is all subtely wrong as Tony seems to say. I don't understand yet. It *seems* to work well and it didn't previously pay much attention to types, it just used INTEGER everywhere. The changes are overall large. Diff the two trees, just in m3back. I actually think my earlier question might be the way to go to dramatically increase testing/coverage/confidence -- cm3 -test64 appends "_TEST64" to the build dir name (maybe maybe not the target name) and sets WORD_SIZE = "64" and BITSIZE(INTEGER) and ADDRESS to 64. I'd even consider something "simpler" where I actually create another complete target, I386_NT_TEST64, including the various entries in m3makefiles, Target.i3, etc. Maybe I can just try this locally with a one line change in Target.m3 though. I'll try that first. > branch and see if the builds and tests succeed. Of course testing with > some other real applications would be great, but we must not delay > endlessly for that. I have to accept releasing with 32bit LONGINT on NT386, due to the large overall change. Hopefully we arrange for more frequent releases somehow. > > - Should maybe improve build/packaging to get NT386-VC80.msi, > > I'd rather not touch the NT386 setup on our virtual machine; I probably won't leave a machine running 24/7. Can I just run the automation manually? Recall Cygwin sshd didn't work adequately at the time. Well, yes, I can always run whatever automation. Somewhat it is a matter of principle of going through the more official more automated process vs. a less official, more error prone, less trusted manual process. Installing additional toolsets on the VM really should work ok, without breaking the existing. Can you try? Maybe this: You create one .msi using the existing process. We'll make sure it is stamped "-VC90" or whatnot. I'll build a whole bunch of others, and they be available as "alternates", and we'll exclude the one corresponding to yours? You provide e.g. cm3-5.8-I386_NT-VC80.msi and I'll provide cm3-5.8-I386_NT-VC20.msi cm3-5.8-I386_NT-VC40.msi cm3-5.8-I386_NT-VC41.msi cm3-5.8-I386_NT-VC42.msi cm3-5.8-I386_NT-VC50.msi cm3-5.8-I386_NT-VC60.msi cm3-5.8-I386_NT-VC70.msi cm3-5.8-I386_NT-VC71.msi cm3-5.8-I386_NT-VC90.msi I realize some of these are quite old and out of use, but it's pretty simple to produce them all. > > - I believe there is "new" divergence in m3front between head and > Is this relevant for the release? Or can we just ignore it? I'd have to look, or Tony should say. To some extent it is tied with if we take NT386 64bit longint. > msi > We still need to add them to the WWW export/display, don't we? Definitely. Sorry of this isn't edited well..took a while already... - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 16 13:32:49 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 12:32:49 +0000 Subject: [M3devel] cvsup Message-ID: I can reproduce the cvsup problem. An important point to consider is that cvsupd forks a server? Maybe that messes up initialization? I'll dig further. I think these steps are complete. I'll test them on a second clean system. You don't need any real data. jay at xlin2:~$ cat cvsupd.cfg *default tag=. *default host=xlin2. *default prefix=/home/jay *default base=/home/jay/var/db *default release=cvs delete use-rel-suffix compress src-all mkdir -p $HOME/var/db mkdir -p $HOME/data/cvsupd pkill cvsupd # cleanup any previous You don't really need any data. cvsup/cvsupd will issue an error in the "working" case, and hang in the non-working case. run the server: jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd 2010.03.16 05:24:25 PDT [22555]: CVSup server started 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 2010.03.16 05:24:25 PDT [22555]: Ready to service requests run the client: /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg Parsing supfile "/home/jay/cvsupd.cfg" Connecting to xlin2. Connected to xlin2. Server software version: 2010-03-05 Negotiating file attribute support Exchanging collection information Server message: Unknown collection "src-all" Establishing multiplexed-mode data connection Running Skipping collection src-all/cvs Shutting down connection to server Finished successfully it is able to talk to the server, and then fails. Ok -- at least it talked to the server. The server then exits too. I think that is correct (read the usage on the -C option). Then try with -C. The bug says -C 2. Let's use -C 1. They behave the same for our purposes. Just add -C 1 to the above server. Run the same client. The client hangs -- it never connects to the server. I reproduced this on LINUXLIBC6 and PPC_LINUX. Maybe more soon. I'm just rebuilding first. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 16 13:38:30 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 13:38:30 +0100 Subject: [M3devel] release status [was something else] In-Reply-To: References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: <20100316133830.8pcec289kcsc0sgk@mail.elegosoft.com> Quoting Jay K : > It is a big change. [...] > I actually think my earlier question might be the way to go to dramatically > increase testing/coverage/confidence -- cm3 -test64 appends "_TEST64" > to the build dir name (maybe maybe not the target name) and sets > WORD_SIZE = "64" and BITSIZE(INTEGER) and ADDRESS to 64. > > I'd even consider something "simpler" where I actually create > another complete > target, I386_NT_TEST64, including the various entries in > m3makefiles, Target.i3, etc. > > Maybe I can just try this locally with a one line change in Target.m3 though. > > I'll try that first. That would be good. [...] > I have to accept releasing with 32bit LONGINT on NT386, due to the large > overall change. > > Hopefully we arrange for more frequent releases somehow. So you think we should not risk the merge and release without the m3back changes now? >> > - Should maybe improve build/packaging to get NT386-VC80.msi, >> >> I'd rather not touch the NT386 setup on our virtual machine; > > I probably won't leave a machine running 24/7. > > Can I just run the automation manually? You can copy and run all the Hudson jobs manually. I'd suggest a parametric setup for the differences though. > Recall Cygwin sshd didn't work adequately at the time. > Well, yes, I can always run whatever automation. Somewhat > it is a matter of principle of going through the more official > more automated > process vs. a less official, more error prone, less trusted > manual process. > Installing additional toolsets on the VM really should work ok, > without breaking > the existing. Can you try? I cannot really do that without GUI access; and I'm afraid there are currently no ressources to setup all those tools (Kay's busy with other work). > Maybe this: You create one .msi using the existing process. We'll make > sure it is stamped "-VC90" or whatnot. > > I'll build a whole bunch of others, and they be available as "alternates", > and we'll exclude the one corresponding to yours? > > You provide e.g. cm3-5.8-I386_NT-VC80.msi and I'll provide > > cm3-5.8-I386_NT-VC20.msi > cm3-5.8-I386_NT-VC40.msi > cm3-5.8-I386_NT-VC41.msi > cm3-5.8-I386_NT-VC42.msi > cm3-5.8-I386_NT-VC50.msi > cm3-5.8-I386_NT-VC60.msi > cm3-5.8-I386_NT-VC70.msi > cm3-5.8-I386_NT-VC71.msi > > cm3-5.8-I386_NT-VC90.msi Do we really need all those? We would not only need the msi installers, but the other packages would have to be compiled and linked with the different tools, too, or they'd be incompatible, wouldn't they? > I realize some of these are quite old and out of use, but it's > pretty simple to produce them all. Of course you can contribute whatever installation archives you can create. I'd rather have a defined set of target platforms and build procedures for the official release though. So I think we should limit our set of supported tool sets. What would you suggest that is most likely to be really useful? > > > - I believe there is "new" divergence in m3front between head and > > Is this relevant for the release? Or can we just ignore it? > > I'd have to look, or Tony should say. > > To some extent it is tied with if we take NT386 64bit longint. > > We still need to add them to the WWW export/display, don't we? > > Definitely. I'll do that. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From hosking at cs.purdue.edu Tue Mar 16 16:07:47 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 11:07:47 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: References: , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu> Message-ID: On 15 Mar 2010, at 21:41, Jay K wrote: > Release status as far as I know: > > - cvsupd hang > Reported over a year ago. Needs investigation. > Can anyone confirm it with a recent version? > And, as Tony asked, with user threads? If anyone gets a snapshot please also make sure to get a backtrace for all the threads. > - NT386 has no 64bit longint in release branch. Partly the point of this question. > > > - Should maybe improve build/packaging to get NT386-VC80.msi, NT386-VC90.msi etc. (not the same as "target/abi alteration via command line" expressed below, though that is a possibility, but separate clean builds for now and/or separate CVS checkouts); really more of a Hudson task now, to have multiple build environments installed and used, I think I already updated the .msi building to append the tag. Olaf? > > > - I believe there is "new" divergence in m3front between head and release -- whatever came along with the TInt changes. Those changes will need bringing over from head to release if Jay's m3back 64-bit support for NT386 are also brought over. I believe there are a few dependencies. > Otherwise I'd have to look through trac. > > > There's maybe some stuff not merged from release to head, but that's ok for release. > (I need to check which branch Randy fixed the examples in.) > I need to re-test the .msis. > > > - Jay > > > > From: hosking at cs.purdue.edu > > To: jay.krell at cornell.edu > > Subject: Re: [M3devel] arbitrary use of LONGINT for testing? target/ABI alteration/generation via command line? > > Date: Mon, 15 Mar 2010 20:00:27 -0500 > > CC: m3devel at elegosoft.com > > > > Let's not right now. We need to get the release out. I suggest a > > freeze on compiler changes for now. Bug fixes only! > > > > Sent from my iPhone > > > > On Mar 15, 2010, at 7:54 PM, Jay K wrote: > > > > > Tony et. al. I'm wondering/fishing if you any ideas, like a cm3 > > > command line switch, that might be used to make INTEGER 64bits to > > > increase coverage/testing of 64bit integer support. > > > > > > > > > The problem is *at least* interfacing with external code/files, if > > > not breaking everything. > > > Maybe also a problem around sizeof(INTEGER) vs. sizeof(ADDRESS). > > > Probably such a setting would grow ADDRESS too. > > > > > > > > > Like, maybe stuff like m3core/unix, m3core/win32 should never use > > > plain INTEGER or LONGINT, but "BITS FOR", to allow this to work? Or > > > maybe <* EXTERNAL *> would be the hint? > > > No, <* EXTERNAL *> isn't adequate, due to the case of callbacks. > > > > > > > > > I'd like to somehow, without too much effort, significantly increase > > > testing of LONGINT. > > > > > > > > > Maybe a wierdo platform NT386_64 that does this just for test > > > purposes? > > > I might try that, if "BITS FOR" is enough to actually let it work > > > (interface with C). > > > (or generally, cm3 -test64 would append "_TEST64" to target name, > > > so we could have LINUXLIBC6_TEST64, I386_DARWIN_TEST64, etc.) > > > > > > > > > I can do some initial experiments along this -test64 line, see if it > > > completely blows up or not. > > > A few minutes thought suggests it should work easily and provide > > > value. > > > > > > > > > (There's probably something similar to try around a 16 bit CHAR. vs. > > > BITS 8 FOR CHAR, but I'm not going there..) > > > > > > > > > Such command line driven target/ABI alterations seem like a somewhat > > > good idea. > > > e.g. cm3 -userthreads would append _USERTHREADS. > > > cm3 -extended80 would use an 80 or 96 bit extended on x86 (96 bits > > > for alignment). > > > -extended128 on ppc > > > etc. > > > > > > > > > cm3 -msvc80 could probe around in enviroment for a Visual C++ 8.0 > > > compiler/linker/header/libraries and append "_MSVC80" to target, > > > else fail. > > > > > > > > > I'll just try -test64 for now. > > > The rest aren't as immediately useful. > > > -userthreads would be my next "favorate". > > > > > > Hm. Instead of -test64, how about -64. > > > If integer is 32 bits, change it to 64 and append _64 (and address). > > > If integer is already 64bits, do nothing. > > > > > > > > > This is quite different than gcc's -m64 though. > > > Maybe not good that way. > > > > > > > > > You can see parallels in several older C compilers. > > > > > > Metrowerks for Mac/68K let you chose integer size and either double > > > or extended size (I forget which). > > > That gave you a cross product set of ABIs, each with their own > > > libraries. > > > > > > > > > Similarly you could chose to issue FPU instructions directly, which > > > might have altered the ABI, or maybe the alternate libraries were > > > just faster. > > > > > > > > > Similarly Microsoft and memory models. Various switches changed the > > > size of a pointer and implied a need for different libraries. > > > > > > > > > Current gcc and ppc floating point sizes I believe is a contemporary > > > example. > > > Even sometimes the affect that -pthread has -- chosing different > > > libraries (unnecessary > > > mess imho, but one that seems to persist on HP-UX.) > > > > > > > > > It's kind of an ugly area, to offer too many choices, produces > > > confusion, but the limited important goal of increasing 64bit > > > integer testing seems worth going here. And increasing userthread > > > testing. People have expressed an interest in higher precision > > > floating point, esp. 80 bit on x86, so this maybe a good way to > > > proceed there also. > > > You could also imagine flags -SSE, -SSE2, etc. > > > > > > > > > It is important to only support a very limited number of these flags. > > > Large cross products are not fun to worry about and test. > > > But for example, with SSE, that nets you several targets all at > > > once, without per-target work. > > > > > > (I lose the term "SSE" loosely. Could be I mean "SSE2" or 3 -- > > > whatever is sufficient to largely replace the stack based x87.) > > > > > > > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 16:21:41 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 11:21:41 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: Ah, yes, one issue about bringing over m3front changes is that it also includes the atomics support. I don't think we want to do this in this release. So, this argues that we hold off on releasing the NT386 64-bit LONGINT support for now. Thoughts? On 16 Mar 2010, at 07:21, Jay K wrote: > Responding to just parts: > > > > > - NT386 has no 64bit longint in release branch. Partly the point of > > > this question. > > > > I thought Randy was going to do some more testing. If the m3back LONGINT > > changes don't break anything else, I'd be for just merging them to the > > > It is a big change. > It is dependent on TInt/TWord changes, though that is flexible. > I could copy and rename them "M3BackInt". > Or we could take them -- they go with m3front changes. > > > Or we could even do a little experiment: The reason I use TInt/TWord > is to get 64bit integers portably to 32bit hosts. Previously INTEGER/Word was used. > I could try using LONGINT/Long instead. > > > However it is still a large change. There are some other unrelated changes, nothing > too significant on its own (I always say), but it adds up: > It isn't messed up, but it is maybe "beyond recognition" compared to the previous. > (as was recently suggested) > > some renaming for, I claim, clarity > > some small optimizations (e.g. using shorter encodings sometimes, like > a byte instead of a dword for constants in instructions, or > equivalent shorter constructs like or reg,-1 instead of mov reg,-1), > > atomics support, actually in very good shape now, but could use more testing and optimization > > "rewrite" of insert/extract to no longer use the tables, I've always > been bothered by those tables > > "rewrite" of set_singleton/set_member to be *much* smaller/faster > set_singleton is just the bts instruction, instead of a function call > set_member is just bt instruction plus carry flag extraction, instead of a function call > > cleanup of the way epilogues are sometimes "patched in" > > Only save/restore non-volatile registers that are actually used. > > Probably other stuff I'm forgetting. > > There's also small related changes in m3objfile: support for outputing more than 4 > bytes at a time, support for "backing up". > Some extra commenting. > Some formating consistency (then always at end of line, else always on its own line). > "Namespace flattening" (from foo import a,b,c). > > I need to test 64bit subrange checking. > The code is a little fishy in that it deals with single registers. > It is an optimization and could probably just be pessimized. > Or maybe easy to fix. > > > There's also the question as to if the TInt use is all subtely wrong as Tony seems to say. > I don't understand yet. > It *seems* to work well and it didn't previously pay much attention to types, it > just used INTEGER everywhere. > > > The changes are overall large. > Diff the two trees, just in m3back. > > > I actually think my earlier question might be the way to go to dramatically > increase testing/coverage/confidence -- cm3 -test64 appends "_TEST64" > to the build dir name (maybe maybe not the target name) and sets > WORD_SIZE = "64" and BITSIZE(INTEGER) and ADDRESS to 64. > I'd even consider something "simpler" where I actually create another complete > target, I386_NT_TEST64, including the various entries in m3makefiles, Target.i3, etc. > > > Maybe I can just try this locally with a one line change in Target.m3 though. > I'll try that first. > > > > branch and see if the builds and tests succeed. Of course testing with > > some other real applications would be great, but we must not delay > > endlessly for that. > > > I have to accept releasing with 32bit LONGINT on NT386, due to the large > overall change. > Hopefully we arrange for more frequent releases somehow. > > > > > > - Should maybe improve build/packaging to get NT386-VC80.msi, > > > > I'd rather not touch the NT386 setup on our virtual machine; > > > I probably won't leave a machine running 24/7. > Can I just run the automation manually? > Recall Cygwin sshd didn't work adequately at the time. > Well, yes, I can always run whatever automation. Somewhat > it is a matter of principle of going through the more official more automated > process vs. a less official, more error prone, less trusted manual process. > > > Installing additional toolsets on the VM really should work ok, without breaking > the existing. Can you try? > > > Maybe this: You create one .msi using the existing process. We'll make > sure it is stamped "-VC90" or whatnot. > I'll build a whole bunch of others, and they be available as "alternates", > and we'll exclude the one corresponding to yours? > You provide e.g. cm3-5.8-I386_NT-VC80.msi and I'll provide > cm3-5.8-I386_NT-VC20.msi > cm3-5.8-I386_NT-VC40.msi > cm3-5.8-I386_NT-VC41.msi > cm3-5.8-I386_NT-VC42.msi > cm3-5.8-I386_NT-VC50.msi > cm3-5.8-I386_NT-VC60.msi > cm3-5.8-I386_NT-VC70.msi > cm3-5.8-I386_NT-VC71.msi > > cm3-5.8-I386_NT-VC90.msi > > > I realize some of these are quite old and out of use, but it's pretty simple to produce > them all. > > > > > - I believe there is "new" divergence in m3front between head and > > Is this relevant for the release? Or can we just ignore it? > > I'd have to look, or Tony should say. > To some extent it is tied with if we take NT386 64bit longint. > > > > msi > > We still need to add them to the WWW export/display, don't we? > > > Definitely. > > > Sorry of this isn't edited well..took a while already... > > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 16:23:34 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 11:23:34 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: <20100316133830.8pcec289kcsc0sgk@mail.elegosoft.com> References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> <20100316133830.8pcec289kcsc0sgk@mail.elegosoft.com> Message-ID: <720A9819-E452-4CEB-A0AA-149B1FF6D88E@cs.purdue.edu> On 16 Mar 2010, at 08:38, Olaf Wagner wrote: > Quoting Jay K : > >> It is a big change. > [...] >> I actually think my earlier question might be the way to go to dramatically >> increase testing/coverage/confidence -- cm3 -test64 appends "_TEST64" >> to the build dir name (maybe maybe not the target name) and sets >> WORD_SIZE = "64" and BITSIZE(INTEGER) and ADDRESS to 64. >> >> I'd even consider something "simpler" where I actually create another complete >> target, I386_NT_TEST64, including the various entries in m3makefiles, Target.i3, etc. >> >> Maybe I can just try this locally with a one line change in Target.m3 though. >> >> I'll try that first. > That would be good. That has pervasive implications. I don't know that there are not large swathes of code that assume that BITSIZE(INTEGER)=BITSIZE(ADDRESS). > > [...] >> I have to accept releasing with 32bit LONGINT on NT386, due to the large >> overall change. Yes, I think that is the way we need to go -- defer 64-bit NT386 LONGINT for now. >> >> Hopefully we arrange for more frequent releases somehow. > > So you think we should not risk the merge and release without the > m3back changes now? Yes. > >>> > - Should maybe improve build/packaging to get NT386-VC80.msi, >>> >>> I'd rather not touch the NT386 setup on our virtual machine; >> >> I probably won't leave a machine running 24/7. >> >> Can I just run the automation manually? > > You can copy and run all the Hudson jobs manually. I'd suggest a > parametric setup for the differences though. > >> Recall Cygwin sshd didn't work adequately at the time. >> Well, yes, I can always run whatever automation. Somewhat >> it is a matter of principle of going through the more official more automated >> process vs. a less official, more error prone, less trusted manual process. > >> Installing additional toolsets on the VM really should work ok, without breaking >> the existing. Can you try? > > I cannot really do that without GUI access; and I'm afraid there are > currently no ressources to setup all those tools (Kay's busy with other > work). > >> Maybe this: You create one .msi using the existing process. We'll make >> sure it is stamped "-VC90" or whatnot. >> >> I'll build a whole bunch of others, and they be available as "alternates", >> and we'll exclude the one corresponding to yours? >> >> You provide e.g. cm3-5.8-I386_NT-VC80.msi and I'll provide >> >> cm3-5.8-I386_NT-VC20.msi >> cm3-5.8-I386_NT-VC40.msi >> cm3-5.8-I386_NT-VC41.msi >> cm3-5.8-I386_NT-VC42.msi >> cm3-5.8-I386_NT-VC50.msi >> cm3-5.8-I386_NT-VC60.msi >> cm3-5.8-I386_NT-VC70.msi >> cm3-5.8-I386_NT-VC71.msi >> >> cm3-5.8-I386_NT-VC90.msi > > Do we really need all those? We would not only need the msi installers, > but the other packages would have to be compiled and linked with the > different tools, too, or they'd be incompatible, wouldn't they? > >> I realize some of these are quite old and out of use, but it's pretty simple to produce them all. > > Of course you can contribute whatever installation archives you can create. > I'd rather have a defined set of target platforms and build procedures > for the official release though. So I think we should limit our > set of supported tool sets. What would you suggest that is most likely > to be really useful? > >> > > - I believe there is "new" divergence in m3front between head and >> > Is this relevant for the release? Or can we just ignore it? >> >> I'd have to look, or Tony should say. >> >> To some extent it is tied with if we take NT386 64bit longint. > >> > We still need to add them to the WWW export/display, don't we? >> >> Definitely. > > I'll do that. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > From wagner at elegosoft.com Tue Mar 16 16:28:42 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 16:28:42 +0100 Subject: [M3devel] release status [was something else] In-Reply-To: References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: <20100316162842.qzx2mmd7cc8kockc@mail.elegosoft.com> Quoting Tony Hosking : > Ah, yes, one issue about bringing over m3front changes is that it > also includes the atomics support. I don't think we want to do this > in this release. > So, this argues that we hold off on releasing the NT386 64-bit > LONGINT support for now. > > Thoughts? It's OK by me. You have convinced me that there are too many dependencies and implications. We should be able to start a new RC build in a few days then. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 16 16:33:29 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 15:33:29 +0000 Subject: [M3devel] cvsup Message-ID: Daniel thank you for the bug report. Thank you for suggesting strace. I used strace and compared working vs. non-working. And started added RTIO everywhere. You can also use cvsup -f to slightly simplify -- one fork instead of two. gdb set follow mode didn't seem to help. I almost have it nailed down. in CVSup, FSServer.m3, this code: FINALLY IF isChild THEN SigHandler.ShutDown(); <== ELSE SigHandler.Unblock(); END; END; which runs fairly early, never returns in the child. It ends up here: PROCEDURE ChangeState(d: Dispatcher; state: State) = (* Ask the dispatcher thread to change to a new state, and wait until it has made the change. *) BEGIN LOCK d.mu DO d.desiredState := state; IF d.state # d.desiredState THEN IF d.state = State.Running THEN (* Send dummy signal 0 to wake up the dispatcher. *) Catch(0); ELSE Thread.Signal(d.changeState); END; WHILE d.state # d.desiredState DO Thread.Wait(d.mu, d.stateChanged); <== this never returns END; END; END; END ChangeState; It's a bit wierd to be mixing fork() and Modula-3 Thread? Or maybe it is ok? See, they are asking another process, from the fork() point of view, to change the state. It does so, but the write is private. Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? sbrk is maybe shared? mmap(anon|private) is not. Right now I have #ifndef apple sbrk #else mmap(anon|shared) #endif and it gets further. Hit an assertion failure in pthread. I'll try again. Cleanup all my RTIO. Maybe this notion of using fork() is just not supportable? In either case...you could paint it as an m3core problem, but ?it won't affect much code?. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: cvsup Date: Tue, 16 Mar 2010 12:32:49 +0000 I can reproduce the cvsup problem. An important point to consider is that cvsupd forks a server? Maybe that messes up initialization? I'll dig further. I think these steps are complete. I'll test them on a second clean system. You don't need any real data. jay at xlin2:~$ cat cvsupd.cfg *default tag=. *default host=xlin2. *default prefix=/home/jay *default base=/home/jay/var/db *default release=cvs delete use-rel-suffix compress src-all mkdir -p $HOME/var/db mkdir -p $HOME/data/cvsupd pkill cvsupd # cleanup any previous You don't really need any data. cvsup/cvsupd will issue an error in the "working" case, and hang in the non-working case. run the server: jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd 2010.03.16 05:24:25 PDT [22555]: CVSup server started 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 2010.03.16 05:24:25 PDT [22555]: Ready to service requests run the client: /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg Parsing supfile "/home/jay/cvsupd.cfg" Connecting to xlin2. Connected to xlin2. Server software version: 2010-03-05 Negotiating file attribute support Exchanging collection information Server message: Unknown collection "src-all" Establishing multiplexed-mode data connection Running Skipping collection src-all/cvs Shutting down connection to server Finished successfully it is able to talk to the server, and then fails. Ok -- at least it talked to the server. The server then exits too. I think that is correct (read the usage on the -C option). Then try with -C. The bug says -C 2. Let's use -C 1. They behave the same for our purposes. Just add -C 1 to the above server. Run the same client. The client hangs -- it never connects to the server. I reproduced this on LINUXLIBC6 and PPC_LINUX. Maybe more soon. I'm just rebuilding first. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 16:43:52 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 11:43:52 -0400 Subject: [M3devel] cvsup In-Reply-To: References: Message-ID: <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> mmap is strongly preferred over sbrk. I'd need to understand the control flow here in and across the processes, but yes, generally, mixing pthreads with fork may require significant care. On 16 Mar 2010, at 11:33, Jay K wrote: > Daniel thank you for the bug report. > Thank you for suggesting strace. I used strace > and compared working vs. non-working. > And started added RTIO everywhere. > You can also use cvsup -f to slightly simplify -- one fork instead of two. > gdb set follow mode didn't seem to help. > > > I almost have it nailed down. > > > in CVSup, FSServer.m3, this code: > > FINALLY > IF isChild THEN > SigHandler.ShutDown(); <== > ELSE > SigHandler.Unblock(); > END; > END; > > > which runs fairly early, never returns in the child. > > > It ends up here: > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > (* Ask the dispatcher thread to change to a new state, and wait until > it has made the change. *) > BEGIN > LOCK d.mu DO > d.desiredState := state; > IF d.state # d.desiredState THEN > IF d.state = State.Running THEN > (* Send dummy signal 0 to wake up the dispatcher. *) > Catch(0); > ELSE > Thread.Signal(d.changeState); > END; > WHILE d.state # d.desiredState DO > Thread.Wait(d.mu, d.stateChanged); <== this never returns > END; > END; > END; > END ChangeState; > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > Or maybe it is ok? > > > See, they are asking another process, from the fork() point of view, to change the state. > It does so, but the write is private. > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > sbrk is maybe shared? mmap(anon|private) is not. > > Right now I have > #ifndef apple > sbrk > #else > mmap(anon|shared) > #endif > > > and it gets further. > Hit an assertion failure in pthread. > I'll try again. > Cleanup all my RTIO. > > > Maybe this notion of using fork() is just not supportable? > > > In either case...you could paint it as an m3core problem, but > ?it won't affect much code?. > > > - Jay > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Subject: cvsup > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > I can reproduce the cvsup problem. > An important point to consider is that cvsupd forks a server? > Maybe that messes up initialization? > I'll dig further. > > > I think these steps are complete. > I'll test them on a second clean system. > You don't need any real data. > > > jay at xlin2:~$ cat cvsupd.cfg > *default tag=. > *default host=xlin2. > *default prefix=/home/jay > *default base=/home/jay/var/db > *default release=cvs delete use-rel-suffix compress > > src-all > > > mkdir -p $HOME/var/db > mkdir -p $HOME/data/cvsupd > pkill cvsupd # cleanup any previous > > > You don't really need any data. > cvsup/cvsupd will issue an error in the "working" case, and > hang in the non-working case. > > > > > run the server: > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > run the client: > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > Parsing supfile "/home/jay/cvsupd.cfg" > Connecting to xlin2. > Connected to xlin2. > Server software version: 2010-03-05 > Negotiating file attribute support > Exchanging collection information > Server message: Unknown collection "src-all" > Establishing multiplexed-mode data connection > Running > Skipping collection src-all/cvs > Shutting down connection to server > Finished successfully > > it is able to talk to the server, and then fails. > Ok -- at least it talked to the server. > > The server then exits too. > I think that is correct (read the usage on the -C option). > > > Then try with -C. > The bug says -C 2. Let's use -C 1. > They behave the same for our purposes. > > > Just add -C 1 to the above server. > Run the same client. > The client hangs -- it never connects to the server. > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > Maybe more soon. > I'm just rebuilding first. > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 16 16:51:42 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 15:51:42 +0000 Subject: [M3devel] release status [was something else] In-Reply-To: <20100316162842.qzx2mmd7cc8kockc@mail.elegosoft.com> References: , , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , , , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com>, , , <20100316162842.qzx2mmd7cc8kockc@mail.elegosoft.com> Message-ID: The "experiment" would also probably have to grow ADDRESS. And then for interop I'd have to say, like HANDLE = BITS 32 FOR ADDRESS, if that is allowed. And then..it gets confused..where would I get "32" from? Compiler would have to truncate/extend pointers. Not sure it is willing to. Another good theoretical route is, indeed widen all the pointers, and then #ifdef the C code to take UINT64 and cast to pointers..and convert structs back/forth..but for Win32 we generally don't have such C code as we do for Posix. Darn. I'll think about it more later but maybe it doesn't really work out. - Jay > Date: Tue, 16 Mar 2010 16:28:42 +0100 > From: wagner at elegosoft.com > To: hosking at cs.purdue.edu > CC: jay.krell at cornell.edu; m3devel at elegosoft.com > Subject: Re: [M3devel] release status [was something else] > > Quoting Tony Hosking : > > > Ah, yes, one issue about bringing over m3front changes is that it > > also includes the atomics support. I don't think we want to do this > > in this release. > > So, this argues that we hold off on releasing the NT386 64-bit > > LONGINT support for now. > > > > Thoughts? > > It's OK by me. You have convinced me that there are too many dependencies > and implications. > > We should be able to start a new RC build in a few days then. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Mar 16 17:10:15 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 16:10:15 +0000 Subject: [M3devel] cvsup In-Reply-To: <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> References: , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> Message-ID: I could have sworn I had not seen hanging with sbrk or map_shared, but I can't get that now. I'll dig a bit more..maybe not today. To be clear, this isn't even the usual fork+exec. This is fork + do work + exit. We can probably fix it by having it create a Modula-3 thread. The first fork (daemonize) is probably fine. It is the per-client one that I think is a problem. I'd like to find a theory as to why it ever worked, maybe see it work, and then fix it differently. Maybe sbrk + user threads? (Can anyone claim to have seen cvsup server work with kernel threads?) - Jay From: hosking at cs.purdue.edu Date: Tue, 16 Mar 2010 11:43:52 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup mmap is strongly preferred over sbrk. I'd need to understand the control flow here in and across the processes, but yes, generally, mixing pthreads with fork may require significant care. On 16 Mar 2010, at 11:33, Jay K wrote: Daniel thank you for the bug report. Thank you for suggesting strace. I used strace and compared working vs. non-working. And started added RTIO everywhere. You can also use cvsup -f to slightly simplify -- one fork instead of two. gdb set follow mode didn't seem to help. I almost have it nailed down. in CVSup, FSServer.m3, this code: FINALLY IF isChild THEN SigHandler.ShutDown(); <== ELSE SigHandler.Unblock(); END; END; which runs fairly early, never returns in the child. It ends up here: PROCEDURE ChangeState(d: Dispatcher; state: State) = (* Ask the dispatcher thread to change to a new state, and wait until it has made the change. *) BEGIN LOCK d.mu DO d.desiredState := state; IF d.state # d.desiredState THEN IF d.state = State.Running THEN (* Send dummy signal 0 to wake up the dispatcher. *) Catch(0); ELSE Thread.Signal(d.changeState); END; WHILE d.state # d.desiredState DO Thread.Wait(d.mu, d.stateChanged); <== this never returns END; END; END; END ChangeState; It's a bit wierd to be mixing fork() and Modula-3 Thread? Or maybe it is ok? See, they are asking another process, from the fork() point of view, to change the state. It does so, but the write is private. Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? sbrk is maybe shared? mmap(anon|private) is not. Right now I have #ifndef apple sbrk #else mmap(anon|shared) #endif and it gets further. Hit an assertion failure in pthread. I'll try again. Cleanup all my RTIO. Maybe this notion of using fork() is just not supportable? In either case...you could paint it as an m3core problem, but ?it won't affect much code?. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: cvsup Date: Tue, 16 Mar 2010 12:32:49 +0000 I can reproduce the cvsup problem. An important point to consider is that cvsupd forks a server? Maybe that messes up initialization? I'll dig further. I think these steps are complete. I'll test them on a second clean system. You don't need any real data. jay at xlin2:~$ cat cvsupd.cfg *default tag=. *default host=xlin2. *default prefix=/home/jay *default base=/home/jay/var/db *default release=cvs delete use-rel-suffix compress src-all mkdir -p $HOME/var/db mkdir -p $HOME/data/cvsupd pkill cvsupd # cleanup any previous You don't really need any data. cvsup/cvsupd will issue an error in the "working" case, and hang in the non-working case. run the server: jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd 2010.03.16 05:24:25 PDT [22555]: CVSup server started 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 2010.03.16 05:24:25 PDT [22555]: Ready to service requests run the client: /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg Parsing supfile "/home/jay/cvsupd.cfg" Connecting to xlin2. Connected to xlin2. Server software version: 2010-03-05 Negotiating file attribute support Exchanging collection information Server message: Unknown collection "src-all" Establishing multiplexed-mode data connection Running Skipping collection src-all/cvs Shutting down connection to server Finished successfully it is able to talk to the server, and then fails. Ok -- at least it talked to the server. The server then exits too. I think that is correct (read the usage on the -C option). Then try with -C. The bug says -C 2. Let's use -C 1. They behave the same for our purposes. Just add -C 1 to the above server. Run the same client. The client hangs -- it never connects to the server. I reproduced this on LINUXLIBC6 and PPC_LINUX. Maybe more soon. I'm just rebuilding first. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 16 17:35:40 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 17:35:40 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> Message-ID: <20100316173540.7cgvahkys44ggw4o@mail.elegosoft.com> Quoting Jay K : > I could have sworn I had not seen hanging with sbrk or map_shared, > but I can't get that now. > > I'll dig a bit more..maybe not today. > > To be clear, this isn't even the usual fork+exec. This is fork + do > work + exit. Well, that's how processes on Unix are usually used for scaling. > We can probably fix it by having it create a Modula-3 thread. Hm. That might work, but wouldn't really be the intention. There should be several server processes each with several threads. > The first fork (daemonize) is probably fine. It is the per-client > one that I think is a problem. > > I'd like to find a theory as to why it ever worked, maybe see it > work, and then fix it differently. > > Maybe sbrk + user threads? > > (Can anyone claim to have seen cvsup server work with kernel threads?) No. I think we always used older binaries at Elego. We should be able to make it work though ;-) Olaf > - Jay > > > > From: hosking at cs.purdue.edu > Date: Tue, 16 Mar 2010 11:43:52 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > mmap is strongly preferred over sbrk. > > > I'd need to understand the control flow here in and across the > processes, but yes, generally, mixing pthreads with fork may require > significant care. > > > > On 16 Mar 2010, at 11:33, Jay K wrote: > > Daniel thank you for the bug report. > Thank you for suggesting strace. I used strace > and compared working vs. non-working. > And started added RTIO everywhere. > You can also use cvsup -f to slightly simplify -- one fork instead of two. > gdb set follow mode didn't seem to help. > > > I almost have it nailed down. > > > in CVSup, FSServer.m3, this code: > > FINALLY > IF isChild THEN > SigHandler.ShutDown(); <== > ELSE > SigHandler.Unblock(); > END; > END; > > > which runs fairly early, never returns in the child. > > > It ends up here: > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > (* Ask the dispatcher thread to change to a new state, and wait until > it has made the change. *) > BEGIN > LOCK d.mu DO > d.desiredState := state; > IF d.state # d.desiredState THEN > IF d.state = State.Running THEN > (* Send dummy signal 0 to wake up the dispatcher. *) > Catch(0); > ELSE > Thread.Signal(d.changeState); > END; > WHILE d.state # d.desiredState DO > Thread.Wait(d.mu, d.stateChanged); <== this never returns > END; > END; > END; > END ChangeState; > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > Or maybe it is ok? > > > See, they are asking another process, from the fork() point of view, > to change the state. > It does so, but the write is private. > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > sbrk is maybe shared? mmap(anon|private) is not. > > Right now I have > #ifndef apple > sbrk > #else > mmap(anon|shared) > #endif > > > and it gets further. > Hit an assertion failure in pthread. > I'll try again. > Cleanup all my RTIO. > > > Maybe this notion of using fork() is just not supportable? > > > In either case...you could paint it as an m3core problem, but > ?it won't affect much code?. > > > - Jay > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Subject: cvsup > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > I can reproduce the cvsup problem. > An important point to consider is that cvsupd forks a server? > Maybe that messes up initialization? > I'll dig further. > > > I think these steps are complete. > I'll test them on a second clean system. > You don't need any real data. > > > jay at xlin2:~$ cat cvsupd.cfg > *default tag=. > *default host=xlin2. > *default prefix=/home/jay > *default base=/home/jay/var/db > *default release=cvs delete use-rel-suffix compress > > src-all > > > mkdir -p $HOME/var/db > mkdir -p $HOME/data/cvsupd > pkill cvsupd # cleanup any previous > > > You don't really need any data. > cvsup/cvsupd will issue an error in the "working" case, and > hang in the non-working case. > > > > > run the server: > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > run the client: > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > Parsing supfile "/home/jay/cvsupd.cfg" > Connecting to xlin2. > Connected to xlin2. > Server software version: 2010-03-05 > Negotiating file attribute support > Exchanging collection information > Server message: Unknown collection "src-all" > Establishing multiplexed-mode data connection > Running > Skipping collection src-all/cvs > Shutting down connection to server > Finished successfully > > it is able to talk to the server, and then fails. > Ok -- at least it talked to the server. > > The server then exits too. > I think that is correct (read the usage on the -C option). > > > Then try with -C. > The bug says -C 2. Let's use -C 1. > They behave the same for our purposes. > > > Just add -C 1 to the above server. > Run the same client. > The client hangs -- it never connects to the server. > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > Maybe more soon. > I'm just rebuilding first. > > > - Jay > > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 16 17:38:00 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 16:38:00 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, Message-ID: User threads is apparently sufficient to let it work -- no change to sbrk vs. mmap. You end up with a wierd mix of shared and not shared data, right? I'll try to make it use pthreads and Modula-3 threads and not fork(). - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Tue, 16 Mar 2010 16:10:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup I could have sworn I had not seen hanging with sbrk or map_shared, but I can't get that now. I'll dig a bit more..maybe not today. To be clear, this isn't even the usual fork+exec. This is fork + do work + exit. We can probably fix it by having it create a Modula-3 thread. The first fork (daemonize) is probably fine. It is the per-client one that I think is a problem. I'd like to find a theory as to why it ever worked, maybe see it work, and then fix it differently. Maybe sbrk + user threads? (Can anyone claim to have seen cvsup server work with kernel threads?) - Jay From: hosking at cs.purdue.edu Date: Tue, 16 Mar 2010 11:43:52 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup mmap is strongly preferred over sbrk. I'd need to understand the control flow here in and across the processes, but yes, generally, mixing pthreads with fork may require significant care. On 16 Mar 2010, at 11:33, Jay K wrote: Daniel thank you for the bug report. Thank you for suggesting strace. I used strace and compared working vs. non-working. And started added RTIO everywhere. You can also use cvsup -f to slightly simplify -- one fork instead of two. gdb set follow mode didn't seem to help. I almost have it nailed down. in CVSup, FSServer.m3, this code: FINALLY IF isChild THEN SigHandler.ShutDown(); <== ELSE SigHandler.Unblock(); END; END; which runs fairly early, never returns in the child. It ends up here: PROCEDURE ChangeState(d: Dispatcher; state: State) = (* Ask the dispatcher thread to change to a new state, and wait until it has made the change. *) BEGIN LOCK d.mu DO d.desiredState := state; IF d.state # d.desiredState THEN IF d.state = State.Running THEN (* Send dummy signal 0 to wake up the dispatcher. *) Catch(0); ELSE Thread.Signal(d.changeState); END; WHILE d.state # d.desiredState DO Thread.Wait(d.mu, d.stateChanged); <== this never returns END; END; END; END ChangeState; It's a bit wierd to be mixing fork() and Modula-3 Thread? Or maybe it is ok? See, they are asking another process, from the fork() point of view, to change the state. It does so, but the write is private. Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? sbrk is maybe shared? mmap(anon|private) is not. Right now I have #ifndef apple sbrk #else mmap(anon|shared) #endif and it gets further. Hit an assertion failure in pthread. I'll try again. Cleanup all my RTIO. Maybe this notion of using fork() is just not supportable? In either case...you could paint it as an m3core problem, but ?it won't affect much code?. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: cvsup Date: Tue, 16 Mar 2010 12:32:49 +0000 I can reproduce the cvsup problem. An important point to consider is that cvsupd forks a server? Maybe that messes up initialization? I'll dig further. I think these steps are complete. I'll test them on a second clean system. You don't need any real data. jay at xlin2:~$ cat cvsupd.cfg *default tag=. *default host=xlin2. *default prefix=/home/jay *default base=/home/jay/var/db *default release=cvs delete use-rel-suffix compress src-all mkdir -p $HOME/var/db mkdir -p $HOME/data/cvsupd pkill cvsupd # cleanup any previous You don't really need any data. cvsup/cvsupd will issue an error in the "working" case, and hang in the non-working case. run the server: jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd 2010.03.16 05:24:25 PDT [22555]: CVSup server started 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 2010.03.16 05:24:25 PDT [22555]: Ready to service requests run the client: /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg Parsing supfile "/home/jay/cvsupd.cfg" Connecting to xlin2. Connected to xlin2. Server software version: 2010-03-05 Negotiating file attribute support Exchanging collection information Server message: Unknown collection "src-all" Establishing multiplexed-mode data connection Running Skipping collection src-all/cvs Shutting down connection to server Finished successfully it is able to talk to the server, and then fails. Ok -- at least it talked to the server. The server then exits too. I think that is correct (read the usage on the -C option). Then try with -C. The bug says -C 2. Let's use -C 1. They behave the same for our purposes. Just add -C 1 to the above server. Run the same client. The client hangs -- it never connects to the server. I reproduced this on LINUXLIBC6 and PPC_LINUX. Maybe more soon. I'm just rebuilding first. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 17:42:22 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 12:42:22 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: References: , , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , , , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com>, , , <20100316162842.qzx2mmd7cc8kockc@mail.elegosoft.com> Message-ID: <1C2120B5-5EC7-4850-8704-B37EFA882493@cs.purdue.edu> I think this would be a huge mess. Not worth pursuing. On 16 Mar 2010, at 11:51, Jay K wrote: > The "experiment" would also probably have to grow ADDRESS. > And then for interop I'd have to say, like HANDLE = BITS 32 FOR ADDRESS, if that is allowed. > And then..it gets confused..where would I get "32" from? > Compiler would have to truncate/extend pointers. Not sure it is willing to. > > Another good theoretical route is, indeed widen all the pointers, and then #ifdef the C code to take UINT64 and cast to pointers..and convert structs back/forth..but for Win32 we generally don't have such C code as we do for Posix. Darn. > I'll think about it more later but maybe it doesn't really work out. > > > - Jay > > > > Date: Tue, 16 Mar 2010 16:28:42 +0100 > > From: wagner at elegosoft.com > > To: hosking at cs.purdue.edu > > CC: jay.krell at cornell.edu; m3devel at elegosoft.com > > Subject: Re: [M3devel] release status [was something else] > > > > Quoting Tony Hosking : > > > > > Ah, yes, one issue about bringing over m3front changes is that it > > > also includes the atomics support. I don't think we want to do this > > > in this release. > > > So, this argues that we hold off on releasing the NT386 64-bit > > > LONGINT support for now. > > > > > > Thoughts? > > > > It's OK by me. You have convinced me that there are too many dependencies > > and implications. > > > > We should be able to start a new RC build in a few days then. > > > > Olaf > > -- > > Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 17:43:41 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 12:43:41 -0400 Subject: [M3devel] cvsup In-Reply-To: References: , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> Message-ID: <68E3ECCB-9F5E-4E4C-B8C2-FF1C14B32448@cs.purdue.edu> I thought I did a year or so back. But not certain. On 16 Mar 2010, at 12:10, Jay K wrote: > I could have sworn I had not seen hanging with sbrk or map_shared, but I can't get that now. > I'll dig a bit more..maybe not today. > To be clear, this isn't even the usual fork+exec. This is fork + do work + exit. > We can probably fix it by having it create a Modula-3 thread. > The first fork (daemonize) is probably fine. It is the per-client one that I think is a problem. > I'd like to find a theory as to why it ever worked, maybe see it work, and then fix it differently. > Maybe sbrk + user threads? > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > - Jay > > From: hosking at cs.purdue.edu > Date: Tue, 16 Mar 2010 11:43:52 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > mmap is strongly preferred over sbrk. > > I'd need to understand the control flow here in and across the processes, but yes, generally, mixing pthreads with fork may require significant care. > > On 16 Mar 2010, at 11:33, Jay K wrote: > > Daniel thank you for the bug report. > Thank you for suggesting strace. I used strace > and compared working vs. non-working. > And started added RTIO everywhere. > You can also use cvsup -f to slightly simplify -- one fork instead of two. > gdb set follow mode didn't seem to help. > > > I almost have it nailed down. > > > in CVSup, FSServer.m3, this code: > > FINALLY > IF isChild THEN > SigHandler.ShutDown(); <== > ELSE > SigHandler.Unblock(); > END; > END; > > > which runs fairly early, never returns in the child. > > > It ends up here: > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > (* Ask the dispatcher thread to change to a new state, and wait until > it has made the change. *) > BEGIN > LOCK d.mu DO > d.desiredState := state; > IF d.state # d.desiredState THEN > IF d.state = State.Running THEN > (* Send dummy signal 0 to wake up the dispatcher. *) > Catch(0); > ELSE > Thread.Signal(d.changeState); > END; > WHILE d.state # d.desiredState DO > Thread.Wait(d.mu, d.stateChanged); <== this never returns > END; > END; > END; > END ChangeState; > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > Or maybe it is ok? > > > See, they are asking another process, from the fork() point of view, to change the state. > It does so, but the write is private. > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > sbrk is maybe shared? mmap(anon|private) is not. > > Right now I have > #ifndef apple > sbrk > #else > mmap(anon|shared) > #endif > > > and it gets further. > Hit an assertion failure in pthread. > I'll try again. > Cleanup all my RTIO. > > > Maybe this notion of using fork() is just not supportable? > > > In either case...you could paint it as an m3core problem, but > ?it won't affect much code?. > > > - Jay > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Subject: cvsup > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > I can reproduce the cvsup problem. > An important point to consider is that cvsupd forks a server? > Maybe that messes up initialization? > I'll dig further. > > > I think these steps are complete. > I'll test them on a second clean system. > You don't need any real data. > > > jay at xlin2:~$ cat cvsupd.cfg > *default tag=. > *default host=xlin2. > *default prefix=/home/jay > *default base=/home/jay/var/db > *default release=cvs delete use-rel-suffix compress > > src-all > > > mkdir -p $HOME/var/db > mkdir -p $HOME/data/cvsupd > pkill cvsupd # cleanup any previous > > > You don't really need any data. > cvsup/cvsupd will issue an error in the "working" case, and > hang in the non-working case. > > > > > run the server: > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > run the client: > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > Parsing supfile "/home/jay/cvsupd.cfg" > Connecting to xlin2. > Connected to xlin2. > Server software version: 2010-03-05 > Negotiating file attribute support > Exchanging collection information > Server message: Unknown collection "src-all" > Establishing multiplexed-mode data connection > Running > Skipping collection src-all/cvs > Shutting down connection to server > Finished successfully > > it is able to talk to the server, and then fails. > Ok -- at least it talked to the server. > > The server then exits too. > I think that is correct (read the usage on the -C option). > > > Then try with -C. > The bug says -C 2. Let's use -C 1. > They behave the same for our purposes. > > > Just add -C 1 to the above server. > Run the same client. > The client hangs -- it never connects to the server. > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > Maybe more soon. > I'm just rebuilding first. > > > - Jay > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 16 18:07:38 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 18:07:38 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, Message-ID: <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> Quoting Jay K : > > User threads is apparently sufficient to let it work -- no change to > sbrk vs. mmap. > > You end up with a wierd mix of shared and not shared data, right? > > I'll try to make it use pthreads and Modula-3 threads and not fork(). I cannot really follow you here. Why should fork not work for threaded processes? The kernel should duplicate all threads and all memory regions, right? I don't seem to get the connection you make to sbrk vs. mmap. Can you explain? Using a thread for the process fork will probably not work contrary to my former statement that it might, as each client process is expected to have multiple threads. And the address spaces of the server processes should be separate for security and robustness reasons, as they are each connected to different clients, and one client's server crash must not impact the other sessions. Or am I misunderstanding what you intend to do? Olaf > - Jay > > > > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Tue, 16 Mar 2010 16:10:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > > > I could have sworn I had not seen hanging with sbrk or map_shared, > but I can't get that now. > I'll dig a bit more..maybe not today. > To be clear, this isn't even the usual fork+exec. This is fork + do > work + exit. > We can probably fix it by having it create a Modula-3 thread. > The first fork (daemonize) is probably fine. It is the per-client > one that I think is a problem. > I'd like to find a theory as to why it ever worked, maybe see it > work, and then fix it differently. > Maybe sbrk + user threads? > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > - Jay > > > > From: hosking at cs.purdue.edu > Date: Tue, 16 Mar 2010 11:43:52 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > mmap is strongly preferred over sbrk. > > > I'd need to understand the control flow here in and across the > processes, but yes, generally, mixing pthreads with fork may require > significant care. > > > > On 16 Mar 2010, at 11:33, Jay K wrote: > > Daniel thank you for the bug report. > Thank you for suggesting strace. I used strace > and compared working vs. non-working. > And started added RTIO everywhere. > You can also use cvsup -f to slightly simplify -- one fork instead of two. > gdb set follow mode didn't seem to help. > > > I almost have it nailed down. > > > in CVSup, FSServer.m3, this code: > > FINALLY > IF isChild THEN > SigHandler.ShutDown(); <== > ELSE > SigHandler.Unblock(); > END; > END; > > > which runs fairly early, never returns in the child. > > > It ends up here: > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > (* Ask the dispatcher thread to change to a new state, and wait until > it has made the change. *) > BEGIN > LOCK d.mu DO > d.desiredState := state; > IF d.state # d.desiredState THEN > IF d.state = State.Running THEN > (* Send dummy signal 0 to wake up the dispatcher. *) > Catch(0); > ELSE > Thread.Signal(d.changeState); > END; > WHILE d.state # d.desiredState DO > Thread.Wait(d.mu, d.stateChanged); <== this never returns > END; > END; > END; > END ChangeState; > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > Or maybe it is ok? > > > See, they are asking another process, from the fork() point of view, > to change the state. > It does so, but the write is private. > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > sbrk is maybe shared? mmap(anon|private) is not. > > Right now I have > #ifndef apple > sbrk > #else > mmap(anon|shared) > #endif > > > and it gets further. > Hit an assertion failure in pthread. > I'll try again. > Cleanup all my RTIO. > > > Maybe this notion of using fork() is just not supportable? > > > In either case...you could paint it as an m3core problem, but > ?it won't affect much code?. > > > - Jay > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Subject: cvsup > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > I can reproduce the cvsup problem. > An important point to consider is that cvsupd forks a server? > Maybe that messes up initialization? > I'll dig further. > > > I think these steps are complete. > I'll test them on a second clean system. > You don't need any real data. > > > jay at xlin2:~$ cat cvsupd.cfg > *default tag=. > *default host=xlin2. > *default prefix=/home/jay > *default base=/home/jay/var/db > *default release=cvs delete use-rel-suffix compress > > src-all > > > mkdir -p $HOME/var/db > mkdir -p $HOME/data/cvsupd > pkill cvsupd # cleanup any previous > > > You don't really need any data. > cvsup/cvsupd will issue an error in the "working" case, and > hang in the non-working case. > > > > > run the server: > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > run the client: > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > Parsing supfile "/home/jay/cvsupd.cfg" > Connecting to xlin2. > Connected to xlin2. > Server software version: 2010-03-05 > Negotiating file attribute support > Exchanging collection information > Server message: Unknown collection "src-all" > Establishing multiplexed-mode data connection > Running > Skipping collection src-all/cvs > Shutting down connection to server > Finished successfully > > it is able to talk to the server, and then fails. > Ok -- at least it talked to the server. > > The server then exits too. > I think that is correct (read the usage on the -C option). > > > Then try with -C. > The bug says -C 2. Let's use -C 1. > They behave the same for our purposes. > > > Just add -C 1 to the above server. > Run the same client. > The client hangs -- it never connects to the server. > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > Maybe more soon. > I'm just rebuilding first. > > > - Jay > > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Tue Mar 16 18:43:40 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 16 Mar 2010 17:43:40 +0000 Subject: [M3devel] cvsup In-Reply-To: <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> Message-ID: With fork, which data is copy-on-write vs. shared-write? mmap has a flag, so it is up to each caller. You end up with a mix. Whether or not you get one forked thread or all of them I believe varies. Solaris has "fork1()". There is no obvious answer. It just goes to show how wierd fork() is. I'm not sure any longer there is a connection to mmap/sbrk. It might be just user threads vs. not. Though there again there is a question as to which data is shared-write vs. copy-on-write. The server should not crash. It is written in an optionally save language, after all. More so, that is I believe a much more larger more difficult fix to apply. Using processes for such separation is deemed overkill, depending on whose story you believe. I'll have to think about it more. - Jay > Date: Tue, 16 Mar 2010 18:07:38 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > Quoting Jay K : > > > > > User threads is apparently sufficient to let it work -- no change to > > sbrk vs. mmap. > > > > You end up with a wierd mix of shared and not shared data, right? > > > > I'll try to make it use pthreads and Modula-3 threads and not fork(). > > I cannot really follow you here. Why should fork not work for > threaded processes? The kernel should duplicate all threads and all > memory regions, right? > > I don't seem to get the connection you make to sbrk vs. mmap. > Can you explain? > > Using a thread for the process fork will probably not work contrary > to my former statement that it might, as each client process is > expected to have multiple threads. > > And the address spaces of the server processes should be separate > for security and robustness reasons, as they are each connected to > different clients, and one client's server crash must not impact the > other sessions. > > Or am I misunderstanding what you intend to do? > > Olaf > > > - Jay > > > > > > > > > > From: jay.krell at cornell.edu > > To: hosking at cs.purdue.edu > > Date: Tue, 16 Mar 2010 16:10:15 +0000 > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] cvsup > > > > > > > > I could have sworn I had not seen hanging with sbrk or map_shared, > > but I can't get that now. > > I'll dig a bit more..maybe not today. > > To be clear, this isn't even the usual fork+exec. This is fork + do > > work + exit. > > We can probably fix it by having it create a Modula-3 thread. > > The first fork (daemonize) is probably fine. It is the per-client > > one that I think is a problem. > > I'd like to find a theory as to why it ever worked, maybe see it > > work, and then fix it differently. > > Maybe sbrk + user threads? > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > > > - Jay > > > > > > > > From: hosking at cs.purdue.edu > > Date: Tue, 16 Mar 2010 11:43:52 -0400 > > To: jay.krell at cornell.edu > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] cvsup > > > > mmap is strongly preferred over sbrk. > > > > > > I'd need to understand the control flow here in and across the > > processes, but yes, generally, mixing pthreads with fork may require > > significant care. > > > > > > > > On 16 Mar 2010, at 11:33, Jay K wrote: > > > > Daniel thank you for the bug report. > > Thank you for suggesting strace. I used strace > > and compared working vs. non-working. > > And started added RTIO everywhere. > > You can also use cvsup -f to slightly simplify -- one fork instead of two. > > gdb set follow mode didn't seem to help. > > > > > > I almost have it nailed down. > > > > > > in CVSup, FSServer.m3, this code: > > > > FINALLY > > IF isChild THEN > > SigHandler.ShutDown(); <== > > ELSE > > SigHandler.Unblock(); > > END; > > END; > > > > > > which runs fairly early, never returns in the child. > > > > > > It ends up here: > > > > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > > (* Ask the dispatcher thread to change to a new state, and wait until > > it has made the change. *) > > BEGIN > > LOCK d.mu DO > > d.desiredState := state; > > IF d.state # d.desiredState THEN > > IF d.state = State.Running THEN > > (* Send dummy signal 0 to wake up the dispatcher. *) > > Catch(0); > > ELSE > > Thread.Signal(d.changeState); > > END; > > WHILE d.state # d.desiredState DO > > Thread.Wait(d.mu, d.stateChanged); <== this never returns > > END; > > END; > > END; > > END ChangeState; > > > > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > > Or maybe it is ok? > > > > > > See, they are asking another process, from the fork() point of view, > > to change the state. > > It does so, but the write is private. > > > > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > > sbrk is maybe shared? mmap(anon|private) is not. > > > > Right now I have > > #ifndef apple > > sbrk > > #else > > mmap(anon|shared) > > #endif > > > > > > and it gets further. > > Hit an assertion failure in pthread. > > I'll try again. > > Cleanup all my RTIO. > > > > > > Maybe this notion of using fork() is just not supportable? > > > > > > In either case...you could paint it as an m3core problem, but > > ?it won't affect much code?. > > > > > > - Jay > > > > > > > > From: jay.krell at cornell.edu > > To: m3devel at elegosoft.com > > Subject: cvsup > > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > > > I can reproduce the cvsup problem. > > An important point to consider is that cvsupd forks a server? > > Maybe that messes up initialization? > > I'll dig further. > > > > > > I think these steps are complete. > > I'll test them on a second clean system. > > You don't need any real data. > > > > > > jay at xlin2:~$ cat cvsupd.cfg > > *default tag=. > > *default host=xlin2. > > *default prefix=/home/jay > > *default base=/home/jay/var/db > > *default release=cvs delete use-rel-suffix compress > > > > src-all > > > > > > mkdir -p $HOME/var/db > > mkdir -p $HOME/data/cvsupd > > pkill cvsupd # cleanup any previous > > > > > > You don't really need any data. > > cvsup/cvsupd will issue an error in the "working" case, and > > hang in the non-working case. > > > > > > > > > > run the server: > > > > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > > > > run the client: > > > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > > > > Parsing supfile "/home/jay/cvsupd.cfg" > > Connecting to xlin2. > > Connected to xlin2. > > Server software version: 2010-03-05 > > Negotiating file attribute support > > Exchanging collection information > > Server message: Unknown collection "src-all" > > Establishing multiplexed-mode data connection > > Running > > Skipping collection src-all/cvs > > Shutting down connection to server > > Finished successfully > > > > it is able to talk to the server, and then fails. > > Ok -- at least it talked to the server. > > > > The server then exits too. > > I think that is correct (read the usage on the -C option). > > > > > > Then try with -C. > > The bug says -C 2. Let's use -C 1. > > They behave the same for our purposes. > > > > > > Just add -C 1 to the above server. > > Run the same client. > > The client hangs -- it never connects to the server. > > > > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > > Maybe more soon. > > I'm just rebuilding first. > > > > > > - Jay > > > > > > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 16 20:32:52 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 16 Mar 2010 15:32:52 -0400 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> Message-ID: fork should still work with pthreads. Why wouldn't it? Unless there is something weird about the semaphore used to start and stop threads? If the child ends up with the same semaphore then that would be a problem. Perhaps we need to ensure it is replicated it in forked children using pthread_atfork? You could try things on a target like OSX where threads are stopped/started natively (i.e., no semaphore being used). Though, on inspection I see that sem_init is called with pshared == false. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 16 Mar 2010, at 13:43, Jay K wrote: > With fork, which data is copy-on-write vs. shared-write? > mmap has a flag, so it is up to each caller. You end up with a mix. > > > Whether or not you get one forked thread or all of them I believe varies. > Solaris has "fork1()". > There is no obvious answer. > It just goes to show how wierd fork() is. > > > I'm not sure any longer there is a connection to mmap/sbrk. > It might be just user threads vs. not. > Though there again there is a question as to which data is > shared-write vs. copy-on-write. > > > The server should not crash. > It is written in an optionally save language, after all. > More so, that is I believe a much more larger more difficult fix to apply. > Using processes for such separation is deemed overkill, depending > on whose story you believe. > I'll have to think about it more. > > > - Jay > > > Date: Tue, 16 Mar 2010 18:07:38 +0100 > > From: wagner at elegosoft.com > > To: m3devel at elegosoft.com > > Subject: Re: [M3devel] cvsup > > > > Quoting Jay K : > > > > > > > > User threads is apparently sufficient to let it work -- no change to > > > sbrk vs. mmap. > > > > > > You end up with a wierd mix of shared and not shared data, right? > > > > > > I'll try to make it use pthreads and Modula-3 threads and not fork(). > > > > I cannot really follow you here. Why should fork not work for > > threaded processes? The kernel should duplicate all threads and all > > memory regions, right? > > > > I don't seem to get the connection you make to sbrk vs. mmap. > > Can you explain? > > > > Using a thread for the process fork will probably not work contrary > > to my former statement that it might, as each client process is > > expected to have multiple threads. > > > > And the address spaces of the server processes should be separate > > for security and robustness reasons, as they are each connected to > > different clients, and one client's server crash must not impact the > > other sessions. > > > > Or am I misunderstanding what you intend to do? > > > > Olaf > > > > > - Jay > > > > > > > > > > > > > > > From: jay.krell at cornell.edu > > > To: hosking at cs.purdue.edu > > > Date: Tue, 16 Mar 2010 16:10:15 +0000 > > > CC: m3devel at elegosoft.com > > > Subject: Re: [M3devel] cvsup > > > > > > > > > > > > I could have sworn I had not seen hanging with sbrk or map_shared, > > > but I can't get that now. > > > I'll dig a bit more..maybe not today. > > > To be clear, this isn't even the usual fork+exec. This is fork + do > > > work + exit. > > > We can probably fix it by having it create a Modula-3 thread. > > > The first fork (daemonize) is probably fine. It is the per-client > > > one that I think is a problem. > > > I'd like to find a theory as to why it ever worked, maybe see it > > > work, and then fix it differently. > > > Maybe sbrk + user threads? > > > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > > > > > - Jay > > > > > > > > > > > > From: hosking at cs.purdue.edu > > > Date: Tue, 16 Mar 2010 11:43:52 -0400 > > > To: jay.krell at cornell.edu > > > CC: m3devel at elegosoft.com > > > Subject: Re: [M3devel] cvsup > > > > > > mmap is strongly preferred over sbrk. > > > > > > > > > I'd need to understand the control flow here in and across the > > > processes, but yes, generally, mixing pthreads with fork may require > > > significant care. > > > > > > > > > > > > On 16 Mar 2010, at 11:33, Jay K wrote: > > > > > > Daniel thank you for the bug report. > > > Thank you for suggesting strace. I used strace > > > and compared working vs. non-working. > > > And started added RTIO everywhere. > > > You can also use cvsup -f to slightly simplify -- one fork instead of two. > > > gdb set follow mode didn't seem to help. > > > > > > > > > I almost have it nailed down. > > > > > > > > > in CVSup, FSServer.m3, this code: > > > > > > FINALLY > > > IF isChild THEN > > > SigHandler.ShutDown(); <== > > > ELSE > > > SigHandler.Unblock(); > > > END; > > > END; > > > > > > > > > which runs fairly early, never returns in the child. > > > > > > > > > It ends up here: > > > > > > > > > PROCEDURE ChangeState(d: Dispatcher; state: State) = > > > (* Ask the dispatcher thread to change to a new state, and wait until > > > it has made the change. *) > > > BEGIN > > > LOCK d.mu DO > > > d.desiredState := state; > > > IF d.state # d.desiredState THEN > > > IF d.state = State.Running THEN > > > (* Send dummy signal 0 to wake up the dispatcher. *) > > > Catch(0); > > > ELSE > > > Thread.Signal(d.changeState); > > > END; > > > WHILE d.state # d.desiredState DO > > > Thread.Wait(d.mu, d.stateChanged); <== this never returns > > > END; > > > END; > > > END; > > > END ChangeState; > > > > > > > > > It's a bit wierd to be mixing fork() and Modula-3 Thread? > > > Or maybe it is ok? > > > > > > > > > See, they are asking another process, from the fork() point of view, > > > to change the state. > > > It does so, but the write is private. > > > > > > > > > Remember...Olaf changed from sbrk to mmap(anon|private) in RTOS.GetMemory()? > > > sbrk is maybe shared? mmap(anon|private) is not. > > > > > > Right now I have > > > #ifndef apple > > > sbrk > > > #else > > > mmap(anon|shared) > > > #endif > > > > > > > > > and it gets further. > > > Hit an assertion failure in pthread. > > > I'll try again. > > > Cleanup all my RTIO. > > > > > > > > > Maybe this notion of using fork() is just not supportable? > > > > > > > > > In either case...you could paint it as an m3core problem, but > > > ?it won't affect much code?. > > > > > > > > > - Jay > > > > > > > > > > > > From: jay.krell at cornell.edu > > > To: m3devel at elegosoft.com > > > Subject: cvsup > > > Date: Tue, 16 Mar 2010 12:32:49 +0000 > > > > > > I can reproduce the cvsup problem. > > > An important point to consider is that cvsupd forks a server? > > > Maybe that messes up initialization? > > > I'll dig further. > > > > > > > > > I think these steps are complete. > > > I'll test them on a second clean system. > > > You don't need any real data. > > > > > > > > > jay at xlin2:~$ cat cvsupd.cfg > > > *default tag=. > > > *default host=xlin2. > > > *default prefix=/home/jay > > > *default base=/home/jay/var/db > > > *default release=cvs delete use-rel-suffix compress > > > > > > src-all > > > > > > > > > mkdir -p $HOME/var/db > > > mkdir -p $HOME/data/cvsupd > > > pkill cvsupd # cleanup any previous > > > > > > > > > You don't really need any data. > > > cvsup/cvsupd will issue an error in the "working" case, and > > > hang in the non-working case. > > > > > > > > > > > > > > > run the server: > > > > > > > > > jay at xlin2:~$ /cm3/bin/cvsupd -b $HOME/data/cvsupd > > > 2010.03.16 05:24:25 PDT [22555]: CVSup server started > > > 2010.03.16 05:24:25 PDT [22555]: Software version: 2010-03-05 10:55:15 > > > 2010.03.16 05:24:25 PDT [22555]: Protocol version: 17.0 > > > 2010.03.16 05:24:25 PDT [22555]: Ready to service requests > > > > > > > > > run the client: > > > > > > /cm3/bin/cvsup -g -L 2 $HOME/cvsupd.cfg > > > > > > > > > Parsing supfile "/home/jay/cvsupd.cfg" > > > Connecting to xlin2. > > > Connected to xlin2. > > > Server software version: 2010-03-05 > > > Negotiating file attribute support > > > Exchanging collection information > > > Server message: Unknown collection "src-all" > > > Establishing multiplexed-mode data connection > > > Running > > > Skipping collection src-all/cvs > > > Shutting down connection to server > > > Finished successfully > > > > > > it is able to talk to the server, and then fails. > > > Ok -- at least it talked to the server. > > > > > > The server then exits too. > > > I think that is correct (read the usage on the -C option). > > > > > > > > > Then try with -C. > > > The bug says -C 2. Let's use -C 1. > > > They behave the same for our purposes. > > > > > > > > > Just add -C 1 to the above server. > > > Run the same client. > > > The client hangs -- it never connects to the server. > > > > > > > > > I reproduced this on LINUXLIBC6 and PPC_LINUX. > > > Maybe more soon. > > > I'm just rebuilding first. > > > > > > > > > - Jay > > > > > > > > > > > > > > -- > > Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Tue Mar 16 21:59:12 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 16 Mar 2010 21:59:12 +0100 Subject: [M3devel] make-deb.py Message-ID: <20100316215912.bvvkuei9wk84w8ks@mail.elegosoft.com> I was wondering why there are no Debian or .msi packages, and now have this trace: + type python python is /usr/local/bin/python + [ xAMD64_FREEBSD = xNT386 ] + python /ad0e/home/wagner/work/cm3/scripts/python/make-deb.py /var/tmp/cm3 set CM3_TARGET=AMD64_FREEBSD set CM3_INSTALL=/var/tmp/cm3 set M3CONFIG=/ad0e/home/wagner/work/cm3/m3-sys/cminstall/src/config-no-install/AMD64_FREEBSD Traceback (most recent call last): File "/ad0e/home/wagner/work/cm3/scripts/python/make-deb.py", line 12, in MakeDebianPackage(sys.argv[1], "/usr/local/cm3") TypeError: MakeDebianPackage() takes exactly 4 arguments (2 given) + mv /var/tmp/cm3.deb /var/tmp/cm3-AMD64_FREEBSD-pre-RC5.deb mv: rename /var/tmp/cm3.deb to /var/tmp/cm3-AMD64_FREEBSD-pre-RC5.deb: No such f It seems the scripts are broken or inconsistent in the release branch. Can you please have a look? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From m3 at sol42.com Tue Mar 16 22:19:26 2010 From: m3 at sol42.com (Daniel Solaz) Date: Tue, 16 Mar 2010 22:19:26 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> Message-ID: On 16 Mar 2010, at 20:32, Tony Hosking wrote: > fork should still work with pthreads. Why wouldn't it? Chapter 6.1 of Butenhof's pthreads book deals with this, and well, it is a mess. According to him only the calling thread exists on return from fork() in the child process, but the other pthread *states* (mutexes, conditions, data keys, etc.) are replicated as well. And I'm sure the details vary from system to system. This is the chapter's first sentence: "Avoid using fork in a threaded program (if you can) unless you intend to exec a new program immediately." Regards. -Daniel From rcolebur at SCIRES.COM Wed Mar 17 01:27:39 2010 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Tue, 16 Mar 2010 20:27:39 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: References: , , <50DDDC7B-65B2-4D51-975F-8EB1E0C064EE@cs.purdue.edu>, , <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: I'm still catching up on my m3devel reading. I was hoping we would put the 64-bit stuff in for NT386. But, I don't want to hold up the release if this is going to be problematic. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Tuesday, March 16, 2010 11:22 AM To: Jay K Cc: m3devel Subject: Re: [M3devel] release status [was something else] Ah, yes, one issue about bringing over m3front changes is that it also includes the atomics support. I don't think we want to do this in this release. So, this argues that we hold off on releasing the NT386 64-bit LONGINT support for now. Thoughts? ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Wed Mar 17 11:35:32 2010 From: hendrik at topoi.pooq.com (hendrik at topoi.pooq.com) Date: Wed, 17 Mar 2010 06:35:32 -0400 Subject: [M3devel] release status [was something else] In-Reply-To: References: <20100316113909.rsj7bqja1wsog888@mail.elegosoft.com> Message-ID: <20100317103531.GB18479@topoi.pooq.com> On Tue, Mar 16, 2010 at 08:27:39PM -0400, Coleburn, Randy wrote: > I'm still catching up on my m3devel reading. > I was hoping we would put the 64-bit stuff in for NT386. > But, I don't want to hold up the release if this is going to be problematic. > Regards, > Randy Judging from the discussions here, the biggest problem in building a release was that it hadn't been done for some time, and the mechanisms for doing it were all rusty. Well, it shouldn't take as long for hte next one. It's mostly well-oiled by now. -- hendrik From jay.krell at cornell.edu Wed Mar 17 08:47:45 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 07:47:45 +0000 Subject: [M3devel] fork/cvsup Message-ID: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Wed Mar 17 10:19:59 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Wed, 17 Mar 2010 10:19:59 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com> Message-ID: <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com> Quoting Daniel Solaz : > On 16 Mar 2010, at 20:32, Tony Hosking wrote: >> fork should still work with pthreads. Why wouldn't it? > Chapter 6.1 of Butenhof's pthreads book deals with this, and well, > it is a mess. According to him only the calling thread exists on > return from fork() in the child process, but the other pthread > *states* (mutexes, conditions, data keys, etc.) are replicated as > well. And I'm sure the details vary from system to system. > This is the chapter's first sentence: "Avoid using fork in a > threaded program (if you can) unless you intend to exec a new > program immediately." Is this about kernel threads or user level threads? Or about a specific implementation of one or the other? I don't know the book, but I always thought about pthreads as being the POSIX specification of threads, and I don't think there's anything in that relating to fork, AFAIR, but that was years ago. Can anyone enlighten us about the POSIX specs and if they define the semantics of fork in presence of threaded processes? Or does it really boil down to the above `avoid it if you can'? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Wed Mar 17 11:07:00 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Wed, 17 Mar 2010 11:07:00 +0100 Subject: [M3devel] Open issues (trac tickets) for cm3 release 5.8 Message-ID: <20100317110700.z1b1sipi80oks8kc@mail.elegosoft.com> There are still a number of open tickets regarding the release in our trac system. Please have a look at https://projects.elego.de/cm3/query?status=resolved&status=reopened&status=assigned&status=analyzed&status=new&status=accepted&group=status&milestone=CM3+release+5.8 I think several of these have already been addressed and can be closed, and others can and should possibly be postponed. Tickets that are assigned to me usually just haven't been assigned to anybody else yet, as I'm the default for that field, so don't ignore them. I can manage these entries, help with information, give you access rights, but haven't the time to do the actual work except in one or two cases. So please try to solve and close as many of the issues as you can before the release. I may also add that analyzing and fixing bugs is always a good starting point for newcomers and those not involved actively so far and just lurking for something to do :-) If you have any questions, don't hesitate to ask me, Olaf PS: Some time ago, there were several requests for trac to coordinate our development. Until now it hasn't really extensively been used; but we should change that. It can really help us to coordinate our work and document changes and their history. It's WiKi can also be used to add various information that is not as formal as tickets and milestones. -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Wed Mar 17 11:42:06 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 10:42:06 +0000 Subject: [M3devel] cvsup In-Reply-To: <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com> References: , , ,,<2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, , , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com> Message-ID: Please start here: http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html "There are at least two serious problems with the semantics of fork() in a multi-threaded program" I've been looking at cvsup a while now. Lots of RTIO.PutText, etc. pthread_atfork usage in RTCollector and ThreadPThread.m3. Reinitialize in the child, mark the gc threads as not being created yet. Been using @M3nogc. It helps. I think I understand why. I have a somewhat wild educated guess as to the problem. It is "fork1" vs. "forkall", sort of. In user threads, when you fork(), you get all the user Modula-3 threads continuing to run in the child. Because their existance is an artifact of a bunch of data that we maintain and gets carried into the new process. With kernel threads (with the exception of using Solaris forkall()), you get just the thread that called fork(). See my reference to the RTCollector/Allocator atfork change, which I show here (not yet commited): PROCEDURE AtForkPrepare() = BEGIN END AtForkPrepare; PROCEDURE AtForkParent() = BEGIN END AtForkParent; PROCEDURE AtForkChild() = VAR r: INTEGER; BEGIN r := Thread.PThreadAtFork(AtForkPrepare, AtForkParent, AtForkChild); <* ASSERT r = 0 *> startedBackground := FALSE; startedForeground := FALSE; END AtForkChild; PROCEDURE Init () = VAR r: INTEGER; BEGIN r := Thread.PThreadAtFork(AtForkPrepare, AtForkParent, AtForkChild); <* ASSERT r = 0 *> Some threads may need to be recreated in the child, some not. The default is not. I think we mainly have to adjust cvsup to recreate its threads. And a little bit of pthread_atfork use in m3core (a bunch of it shown believe, and reinitialization in ThreadPThread.m3). Like the Apple/Darwin warnings, I don't think most libraries can be considered "fork safe". That is you can fork, but only if you exec soon thereafter. Making code "fork safe" I believe equates to reviewing it a bunch and using pthread_atfork selectively. One thing to watch for is if the library creates any worker threads during "initialization" or even "on demand" -- to either recreate them in the child, or note that they don't exist in the child. We can make m3core fork safe, I guess, to satisfy cvsup. Maybe write the code to duplicate all threads in a child, subject to some configuration setting? Something like @M3forkall, but it'd be something that e.g. cvsup would specify when it builds, and then user wouldn't have to list it anywhere. Kind of like the flags for incremental and vm gc that get recorded by the compiler. Anyway, let me see about recreating cvsup's threads manually. Probably by using the Thread.PThreadAtFork I added. I think the initial hang I described fits my theory -- the dispatcher thread doesn't exist in the child, so hang. And then when I avoid that with a local edit I get a hang later. I compared without -C and I see other stuff happening..I think on other threads that fork() is abandoning.. Thread.PThreadAtFork is wierd, but I think reasonable. It will do nothing on user threads and Win32. It will only do anything on PThreads. Libraries can use it to achieve compatibility with pthreads programs that use fork. - Jay > Date: Wed, 17 Mar 2010 10:19:59 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > Quoting Daniel Solaz : > > > On 16 Mar 2010, at 20:32, Tony Hosking wrote: > >> fork should still work with pthreads. Why wouldn't it? > > Chapter 6.1 of Butenhof's pthreads book deals with this, and well, > > it is a mess. According to him only the calling thread exists on > > return from fork() in the child process, but the other pthread > > *states* (mutexes, conditions, data keys, etc.) are replicated as > > well. And I'm sure the details vary from system to system. > > This is the chapter's first sentence: "Avoid using fork in a > > threaded program (if you can) unless you intend to exec a new > > program immediately." > > Is this about kernel threads or user level threads? Or about a specific > implementation of one or the other? > > I don't know the book, but I always thought about pthreads as being the > POSIX specification of threads, and I don't think there's anything > in that relating to fork, AFAIR, but that was years ago. > > Can anyone enlighten us about the POSIX specs and if they define the > semantics of fork in presence of threaded processes? Or does it really > boil down to the above `avoid it if you can'? > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 11:50:31 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 10:50:31 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , ,,,,<2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>,,, , , ,,, , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, , , , , , , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com>, Message-ID: Here's a shorter more direct version. Please start here: http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html "There are at least two serious problems with the semantics of fork() in a multi-threaded program" I have a good theory as to the problem and the fix. Problem: With user threads, when you fork(), all the threads keep running in the new child. Because the data that causes them to exist is carried forward. With kernel threads, they don't, just the caller of fork(). Fix: A few small uses of pthread_atfork both in m3core and cvsup. Abstracted behind Thread.PThreadAtFork that does nothing for user threads and Win32. They would do things like "reinitialize globals" and "recreate worker threads". You can't just call all the module initializers over. That would ultimately I believe reset too much data. ? It might be possible, though, to setjmp, run the module initializers, longjmp. Something like, perhaps, a flag to RTLinker.InitRuntime that causes it to longjmp instead of calling main. However this would still e.g. give the initial thread the wrong stackbase and mess up garbage collection. I'm much more keen on selective use of pthread_atfork, in m3core and cvsup. A more conventional approach is the program to rerun itself with some flags that "push" it fast to "resume" point, but that somewhat defeats the purpose of the fork + do work model -- cheap reuse of already established state. Lingering problem: Any libraries that create worker threads need to use Thread.PThreadAtFork in order to be compatible with the fork + do more work pattern. fork + exec is fine. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 12:02:02 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 11:02:02 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , , , , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , , , , , , , , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, ,,, ,,, ,,, , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com>, , , Message-ID: Another good option might be "RecreateThreadsAfterFork()" that cvsup can call. I'm trying that. If that doesn't work, I'll track down all of cvsup's thread creates. - Jay From: jay.krell at cornell.edu To: wagner at elegosoft.com; m3devel at elegosoft.com Date: Wed, 17 Mar 2010 10:50:31 +0000 Subject: Re: [M3devel] cvsup Here's a shorter more direct version. Please start here: http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html "There are at least two serious problems with the semantics of fork() in a multi-threaded program" I have a good theory as to the problem and the fix. Problem: With user threads, when you fork(), all the threads keep running in the new child. Because the data that causes them to exist is carried forward. With kernel threads, they don't, just the caller of fork(). Fix: A few small uses of pthread_atfork both in m3core and cvsup. Abstracted behind Thread.PThreadAtFork that does nothing for user threads and Win32. They would do things like "reinitialize globals" and "recreate worker threads". You can't just call all the module initializers over. That would ultimately I believe reset too much data. ? It might be possible, though, to setjmp, run the module initializers, longjmp. Something like, perhaps, a flag to RTLinker.InitRuntime that causes it to longjmp instead of calling main. However this would still e.g. give the initial thread the wrong stackbase and mess up garbage collection. I'm much more keen on selective use of pthread_atfork, in m3core and cvsup. A more conventional approach is the program to rerun itself with some flags that "push" it fast to "resume" point, but that somewhat defeats the purpose of the fork + do work model -- cheap reuse of already established state. Lingering problem: Any libraries that create worker threads need to use Thread.PThreadAtFork in order to be compatible with the fork + do more work pattern. fork + exec is fine. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 12:10:43 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 11:10:43 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , , , , , ,,<2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , ,,, , , , , ,,, , ,,, <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, , , , , , , , , , , , , , , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com>, , , , , , Message-ID: For that matter, we can just provide Thread.ForkAll. special implementation on pthreads. Just call Unix.fork() on user threads. No implementation for Win32. - Jay From: jay.krell at cornell.edu To: wagner at elegosoft.com; m3devel at elegosoft.com Date: Wed, 17 Mar 2010 11:02:02 +0000 Subject: Re: [M3devel] cvsup Another good option might be "RecreateThreadsAfterFork()" that cvsup can call. I'm trying that. If that doesn't work, I'll track down all of cvsup's thread creates. - Jay From: jay.krell at cornell.edu To: wagner at elegosoft.com; m3devel at elegosoft.com Date: Wed, 17 Mar 2010 10:50:31 +0000 Subject: Re: [M3devel] cvsup Here's a shorter more direct version. Please start here: http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html "There are at least two serious problems with the semantics of fork() in a multi-threaded program" I have a good theory as to the problem and the fix. Problem: With user threads, when you fork(), all the threads keep running in the new child. Because the data that causes them to exist is carried forward. With kernel threads, they don't, just the caller of fork(). Fix: A few small uses of pthread_atfork both in m3core and cvsup. Abstracted behind Thread.PThreadAtFork that does nothing for user threads and Win32. They would do things like "reinitialize globals" and "recreate worker threads". You can't just call all the module initializers over. That would ultimately I believe reset too much data. ? It might be possible, though, to setjmp, run the module initializers, longjmp. Something like, perhaps, a flag to RTLinker.InitRuntime that causes it to longjmp instead of calling main. However this would still e.g. give the initial thread the wrong stackbase and mess up garbage collection. I'm much more keen on selective use of pthread_atfork, in m3core and cvsup. A more conventional approach is the program to rerun itself with some flags that "push" it fast to "resume" point, but that somewhat defeats the purpose of the fork + do work model -- cheap reuse of already established state. Lingering problem: Any libraries that create worker threads need to use Thread.PThreadAtFork in order to be compatible with the fork + do more work pattern. fork + exec is fine. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Wed Mar 17 12:28:50 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Wed, 17 Mar 2010 12:28:50 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , ,,,,<2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>,,, , , ,,, , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, , , , , , , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com>, Message-ID: <20100317122850.gd98fzkfs0kokgs4@mail.elegosoft.com> Quoting Jay K : > Here's a shorter more direct version. > > Please start here: > http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html > "There are at least two serious problems with the semantics of > fork() in a multi-threaded program" > > I have a good theory as to the problem and the fix. > > Problem: > > With user threads, when you fork(), all the threads keep running in > the new child. > > Because the data that causes them to exist is carried forward. > > With kernel threads, they don't, just the caller of fork(). This is where I seem to have been wrong. I always thought that fork() would indeed duplicate all threads and their execution states. > Fix: > > A few small uses of pthread_atfork both in m3core and cvsup. > > Abstracted behind Thread.PThreadAtFork that does nothing for user > threads and Win32. > > They would do things like "reinitialize globals" and "recreate > worker threads". This sounds reasonable, but is of course not trivial. I think we should at least guarantee that the M3 runtime threads that were running are available again and will do their work. I'm not sure if cvsupd itself will need application extensions, or if the threads handling the streaming protocol are created after the fork. John Polstra used to be on this list and may be able to help. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Wed Mar 17 13:41:23 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 12:41:23 +0000 Subject: [M3devel] cvsup In-Reply-To: <20100317122850.gd98fzkfs0kokgs4@mail.elegosoft.com> References: , , , , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , , , , , , , , , , <20100316180738.f739cbxpes4800kc@mail.elegosoft.com>, , , , , , , , <20100317101959.cnqg4db5msgsccw8@mail.elegosoft.com>, , , <20100317122850.gd98fzkfs0kokgs4@mail.elegosoft.com> Message-ID: Olaf, also search the web for fork1, forkall. Solaris has them. You were right for Modula-3 user threads. But not for Win32 or pthreads. I tried something by accident..and it seemed to work. In my attempts to write PThread.ForkAll, which would fork() and then in the child initialize and recreate the threads of the parent (except for the current thread), I commented out the actual creating, leaving in the initialization..it oddly, it worked. I probably had the "ShutDown" hack in cvsup (direct call instead of through the cvsup dispatcher). Perhaps this works because my cvsup test case is too small and doesn't require its other threads. Or maybe it suffices. Still, there are two approaches to a fix. They are both not difficult. One is to provide the overarching function: Thread.ForkAll(). Cvsup would just call that instead of Unix.fork. The other is to provide pthread_atfork wrapper, use it internally a little bit, and have cvsup use it as well to recreate its threads. But I'm not sure it needs to. Modula-3 wouldn't actually have to recreate any threads, just set the booleans in RTCollector to false indicating they haven't been created. Could be that cvsup doesn't require any/many threads to flow from server to forked server, maybe just the one that it tears down right away. - Jay > Date: Wed, 17 Mar 2010 12:28:50 +0100 > From: wagner at elegosoft.com > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] cvsup > > Quoting Jay K : > > > Here's a shorter more direct version. > > > > Please start here: > > http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html > > "There are at least two serious problems with the semantics of > > fork() in a multi-threaded program" > > > > I have a good theory as to the problem and the fix. > > > > Problem: > > > > With user threads, when you fork(), all the threads keep running in > > the new child. > > > > Because the data that causes them to exist is carried forward. > > > > With kernel threads, they don't, just the caller of fork(). > > This is where I seem to have been wrong. I always thought that fork() > would indeed duplicate all threads and their execution states. > > > Fix: > > > > A few small uses of pthread_atfork both in m3core and cvsup. > > > > Abstracted behind Thread.PThreadAtFork that does nothing for user > > threads and Win32. > > > > They would do things like "reinitialize globals" and "recreate > > worker threads". > > This sounds reasonable, but is of course not trivial. > I think we should at least guarantee that the M3 runtime threads > that were running are available again and will do their work. > > I'm not sure if cvsupd itself will need application extensions, or if > the threads handling the streaming protocol are created after the fork. > > John Polstra used to be on this list and may be able to help. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 17 14:28:14 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 17 Mar 2010 09:28:14 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: References: Message-ID: Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such time as one of the exec functions is > called. All APIs, including global data symbols, in any framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 15:01:15 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 14:01:15 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: References: , Message-ID: Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 16:39:59 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 15:39:59 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , Message-ID: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 17 19:01:31 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 17 Mar 2010 14:01:31 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , Message-ID: <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu> Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: > I have something working on Solaris now. > More details after testing on Linux and Darwin. > > - Jay > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Exec what? > You'd have to change the code to carefully reach the same place. > > - Jay > > Subject: Re: [M3devel] fork/cvsup > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 09:28:14 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not just exec in the child? > > On 17 Mar 2010, at 03:47, Jay K wrote: > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such time as one of the exec functions is > called. All APIs, including global data symbols, in any framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 19:13:45 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 18:13:45 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu> References: , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu> Message-ID: --- bad news: It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. Restart manually. Works again. Around 9 times. Then hangs again. That is on Linux/x86 and Solaris/sparc. Doesn't work at all on Mac/amd64, just hangs. --- sketch: m3core uses pthread_atfork to selectively reinitialize Mainly to only have one thread. common Thread.PThreadAtFork is provided for others to do the same It is deliberately in a portable interface. Thread.ReforkThreadAfterProcessFork Is provided for users to restart threads from their child AtFork hander. This is used by the allocator/collector. Thread.ForkProcessAndAllThreads() Is used by "lazy" clients who want to restart all their threads but didn't keep track of them. The runtime can do it for them. This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads or not, depending on if they need their threads restarted. The runtime takes care of its threads either way. --- What'd I'd written up: attached works typically 9 times on Linux and Solaris before server hangs again. No improvement on Darwin, just hangs. Can't see much in debuggers for some reason. There is extra allowance in the m3core change such that users of fork + do work (as opposed to fork + exec) may or may not call ForkAll, depending on if they feel a need for their own threads to be recreated, and if they've kept track of how to recreate them, or just rely on the runtime to know all the threads. There are three runtime threads that are sometimes created in the parent, and if so, recreated in the child. background collector, foreground collector, weak ref thread I'll try to poke at it some more. I'm not sure what is the best way to suspend all threads. I tried a few differnt ways. SuspendOthers LockHeap pthread_mutex_lock various combinations It is deliberate that pthread specific code is in common/Thread.i3. That way code can be portable, at least among the two Posix thread implementations. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:31 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: m3core_atfork.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cvsup_forkall.txt URL: From hosking at cs.purdue.edu Wed Mar 17 19:30:47 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 17 Mar 2010 14:30:47 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu> Message-ID: <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu> I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 14:13, Jay K wrote: > --- > bad news: > It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. > Restart manually. Works again. Around 9 times. Then hangs again. > That is on Linux/x86 and Solaris/sparc. > Doesn't work at all on Mac/amd64, just hangs. > > --- > sketch: > m3core uses pthread_atfork to selectively reinitialize > Mainly to only have one thread. > > > common Thread.PThreadAtFork is provided for others to do the same > It is deliberately in a portable interface. > > > Thread.ReforkThreadAfterProcessFork > Is provided for users to restart threads from their child AtFork hander. > This is used by the allocator/collector. > > > Thread.ForkProcessAndAllThreads() > Is used by "lazy" clients who want to restart all their threads > but didn't keep track of them. The runtime can do it for them. > > > This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads > or not, depending on if they need their threads restarted. > The runtime takes care of its threads either way. > > > --- > What'd I'd written up: > > attached works typically 9 times on Linux and Solaris > before server hangs again. > > > No improvement on Darwin, just hangs. > Can't see much in debuggers for some reason. > > > There is extra allowance in the m3core change such > that users of fork + do work (as opposed to fork + exec) > may or may not call ForkAll, depending on if they > feel a need for their own threads to be recreated, > and if they've kept track of how to recreate them, > or just rely on the runtime to know all the threads. > > > There are three runtime threads that are sometimes > created in the parent, and if so, recreated in the child. > background collector, foreground collector, weak ref thread > > > I'll try to poke at it some more. > > > I'm not sure what is the best way to suspend all threads. > I tried a few differnt ways. > SuspendOthers > LockHeap > pthread_mutex_lock > various combinations > > > It is deliberate that pthread specific code is in common/Thread.i3. > That way code can be portable, at least among the two Posix thread implementations. > > > - Jay > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:31 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Can you sketch the approach you've taken? > > > On 17 Mar 2010, at 11:39, Jay K wrote: > > I have something working on Solaris now. > More details after testing on Linux and Darwin. > > - Jay > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Exec what? > You'd have to change the code to carefully reach the same place. > > - Jay > > Subject: Re: [M3devel] fork/cvsup > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 09:28:14 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not just exec in the child? > > On 17 Mar 2010, at 03:47, Jay K wrote: > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such time as one of the exec functions is > called. All APIs, including global data symbols, in any framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 23:05:25 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 22:05:25 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu> References: , , ,,, , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu> Message-ID: Tony, I don't know. Here is some "argument', but I'm not sure. Adding threads does something different. Such threads would share mutation to global state. I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:30:47 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 14:13, Jay K wrote: --- bad news: It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. Restart manually. Works again. Around 9 times. Then hangs again. That is on Linux/x86 and Solaris/sparc. Doesn't work at all on Mac/amd64, just hangs. --- sketch: m3core uses pthread_atfork to selectively reinitialize Mainly to only have one thread. common Thread.PThreadAtFork is provided for others to do the same It is deliberately in a portable interface. Thread.ReforkThreadAfterProcessFork Is provided for users to restart threads from their child AtFork hander. This is used by the allocator/collector. Thread.ForkProcessAndAllThreads() Is used by "lazy" clients who want to restart all their threads but didn't keep track of them. The runtime can do it for them. This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads or not, depending on if they need their threads restarted. The runtime takes care of its threads either way. --- What'd I'd written up: attached works typically 9 times on Linux and Solaris before server hangs again. No improvement on Darwin, just hangs. Can't see much in debuggers for some reason. There is extra allowance in the m3core change such that users of fork + do work (as opposed to fork + exec) may or may not call ForkAll, depending on if they feel a need for their own threads to be recreated, and if they've kept track of how to recreate them, or just rely on the runtime to know all the threads. There are three runtime threads that are sometimes created in the parent, and if so, recreated in the child. background collector, foreground collector, weak ref thread I'll try to poke at it some more. I'm not sure what is the best way to suspend all threads. I tried a few differnt ways. SuspendOthers LockHeap pthread_mutex_lock various combinations It is deliberate that pthread specific code is in common/Thread.i3. That way code can be portable, at least among the two Posix thread implementations. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:31 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 17 23:18:37 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 22:18:37 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: References: , ,,,,,,, , , ,,, , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, Message-ID: Maybe an easier approach is to offer a way in quake to select user threads, that then "sticks" automatically through runtime? That either implies build_standalone, or provides a differently named m3core.so? Replacing m3core.so doesn't work, and wouldn't solve this problem. Presumably one would want cvsup to coexist with non-user threads programs. You have expressed not liking these ideas in the past though. It wouldn't necessarily cost even a function pointer indirection for pthreads. That is, it wouldn't necessarily be an @M3userthreads runtime choice. The choice would be made at compile time and the appropriate functions called directly. Well, granted, the easiest way to implement that is "directly" through other functions. Programs that need "fork all" semantics could use user threads. We could alter the types slightly I think so they have the same fingerprints. So that everything wouldn't have to be recompiled, so that it'd be a build time replacement of one .lib or the other. Ie. first fix it so that swapping .so files works. And then make user_threads imply build_standalone. Does that make sense? To repeat: first adjust the two Thread.T to match, if that is possible. Not their code, just the types. And then build m3core twice. And have user threads imply build_standalone. It'd probably just be a matter of adding a few unused fields. I think I'll first try the "limited atfork" idea. That code is in hand already at least to start. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 22:05:25 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Tony, I don't know. Here is some "argument', but I'm not sure. Adding threads does something different. Such threads would share mutation to global state. I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:30:47 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 14:13, Jay K wrote: --- bad news: It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. Restart manually. Works again. Around 9 times. Then hangs again. That is on Linux/x86 and Solaris/sparc. Doesn't work at all on Mac/amd64, just hangs. --- sketch: m3core uses pthread_atfork to selectively reinitialize Mainly to only have one thread. common Thread.PThreadAtFork is provided for others to do the same It is deliberately in a portable interface. Thread.ReforkThreadAfterProcessFork Is provided for users to restart threads from their child AtFork hander. This is used by the allocator/collector. Thread.ForkProcessAndAllThreads() Is used by "lazy" clients who want to restart all their threads but didn't keep track of them. The runtime can do it for them. This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads or not, depending on if they need their threads restarted. The runtime takes care of its threads either way. --- What'd I'd written up: attached works typically 9 times on Linux and Solaris before server hangs again. No improvement on Darwin, just hangs. Can't see much in debuggers for some reason. There is extra allowance in the m3core change such that users of fork + do work (as opposed to fork + exec) may or may not call ForkAll, depending on if they feel a need for their own threads to be recreated, and if they've kept track of how to recreate them, or just rely on the runtime to know all the threads. There are three runtime threads that are sometimes created in the parent, and if so, recreated in the child. background collector, foreground collector, weak ref thread I'll try to poke at it some more. I'm not sure what is the best way to suspend all threads. I tried a few differnt ways. SuspendOthers LockHeap pthread_mutex_lock various combinations It is deliberate that pthread specific code is in common/Thread.i3. That way code can be portable, at least among the two Posix thread implementations. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:31 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 17 23:32:44 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 17 Mar 2010 18:32:44 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , , , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu> Message-ID: <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> What are the expectations in the cvsup child regarding the threads it inherits? On 17 Mar 2010, at 18:05, Jay K wrote: > Tony, I don't know. > Here is some "argument', but I'm not sure. > > > Adding threads does something different. Such threads would share mutation to global state. > I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. > > I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave > people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. > > > There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:30:47 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 17 Mar 2010, at 14:13, Jay K wrote: > > --- > bad news: > It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. > Restart manually. Works again. Around 9 times. Then hangs again. > That is on Linux/x86 and Solaris/sparc. > Doesn't work at all on Mac/amd64, just hangs. > > --- > sketch: > m3core uses pthread_atfork to selectively reinitialize > Mainly to only have one thread. > > > common Thread.PThreadAtFork is provided for others to do the same > It is deliberately in a portable interface. > > > Thread.ReforkThreadAfterProcessFork > Is provided for users to restart threads from their child AtFork hander. > This is used by the allocator/collector. > > > Thread.ForkProcessAndAllThreads() > Is used by "lazy" clients who want to restart all their threads > but didn't keep track of them. The runtime can do it for them. > > > This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads > or not, depending on if they need their threads restarted. > The runtime takes care of its threads either way. > > > --- > What'd I'd written up: > > attached works typically 9 times on Linux and Solaris > before server hangs again. > > > No improvement on Darwin, just hangs. > Can't see much in debuggers for some reason. > > > There is extra allowance in the m3core change such > that users of fork + do work (as opposed to fork + exec) > may or may not call ForkAll, depending on if they > feel a need for their own threads to be recreated, > and if they've kept track of how to recreate them, > or just rely on the runtime to know all the threads. > > > There are three runtime threads that are sometimes > created in the parent, and if so, recreated in the child. > background collector, foreground collector, weak ref thread > > > I'll try to poke at it some more. > > > I'm not sure what is the best way to suspend all threads. > I tried a few differnt ways. > SuspendOthers > LockHeap > pthread_mutex_lock > various combinations > > > It is deliberate that pthread specific code is in common/Thread.i3. > That way code can be portable, at least among the two Posix thread implementations. > > > - Jay > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:31 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Can you sketch the approach you've taken? > > > On 17 Mar 2010, at 11:39, Jay K wrote: > > I have something working on Solaris now. > More details after testing on Linux and Darwin. > > - Jay > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Exec what? > You'd have to change the code to carefully reach the same place. > > - Jay > > Subject: Re: [M3devel] fork/cvsup > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 09:28:14 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not just exec in the child? > > On 17 Mar 2010, at 03:47, Jay K wrote: > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such time as one of the exec functions is > called. All APIs, including global data symbols, in any framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 17 23:40:17 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 17 Mar 2010 18:40:17 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , , , , , , , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, Message-ID: <5EBE34A1-B22E-4E03-AB59-5A7E4ABFAAA9@cs.purdue.edu> That makes little sense. Surely the cvsup server would want to run with proper multi-threading? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 18:18, Jay K wrote: > Maybe an easier approach is to offer a way in quake to select user threads, > that then "sticks" automatically through runtime? > That either implies build_standalone, or provides a differently named m3core.so? > Replacing m3core.so doesn't work, and wouldn't solve this problem. Presumably > one would want cvsup to coexist with non-user threads programs. > > > You have expressed not liking these ideas in the past though. > > > It wouldn't necessarily cost even a function pointer indirection for pthreads. > That is, it wouldn't necessarily be an @M3userthreads runtime choice. > > > The choice would be made at compile time and the appropriate functions called directly. > Well, granted, the easiest way to implement that is "directly" through other functions. > > > Programs that need "fork all" semantics could use user threads. > > > We could alter the types slightly I think so they have the same fingerprints. > So that everything wouldn't have to be recompiled, so that it'd be a build time > replacement of one .lib or the other. Ie. first fix it so that swapping .so files > works. And then make user_threads imply build_standalone. > Does that make sense? > To repeat: first adjust the two Thread.T to match, if that is possible. > Not their code, just the types. > And then build m3core twice. > And have user threads imply build_standalone. > > > It'd probably just be a matter of adding a few unused fields. > > > I think I'll first try the "limited atfork" idea. > That code is in hand already at least to start. > > > - Jay > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 22:05:25 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Tony, I don't know. > Here is some "argument', but I'm not sure. > > > Adding threads does something different. Such threads would share mutation to global state. > I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. > > I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave > people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. > > > There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:30:47 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On 17 Mar 2010, at 14:13, Jay K wrote: > > --- > bad news: > It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. > Restart manually. Works again. Around 9 times. Then hangs again. > That is on Linux/x86 and Solaris/sparc. > Doesn't work at all on Mac/amd64, just hangs. > > --- > sketch: > m3core uses pthread_atfork to selectively reinitialize > Mainly to only have one thread. > > > common Thread.PThreadAtFork is provided for others to do the same > It is deliberately in a portable interface. > > > Thread.ReforkThreadAfterProcessFork > Is provided for users to restart threads from their child AtFork hander. > This is used by the allocator/collector. > > > Thread.ForkProcessAndAllThreads() > Is used by "lazy" clients who want to restart all their threads > but didn't keep track of them. The runtime can do it for them. > > > This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads > or not, depending on if they need their threads restarted. > The runtime takes care of its threads either way. > > > --- > What'd I'd written up: > > attached works typically 9 times on Linux and Solaris > before server hangs again. > > > No improvement on Darwin, just hangs. > Can't see much in debuggers for some reason. > > > There is extra allowance in the m3core change such > that users of fork + do work (as opposed to fork + exec) > may or may not call ForkAll, depending on if they > feel a need for their own threads to be recreated, > and if they've kept track of how to recreate them, > or just rely on the runtime to know all the threads. > > > There are three runtime threads that are sometimes > created in the parent, and if so, recreated in the child. > background collector, foreground collector, weak ref thread > > > I'll try to poke at it some more. > > > I'm not sure what is the best way to suspend all threads. > I tried a few differnt ways. > SuspendOthers > LockHeap > pthread_mutex_lock > various combinations > > > It is deliberate that pthread specific code is in common/Thread.i3. > That way code can be portable, at least among the two Posix thread implementations. > > > - Jay > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:31 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Can you sketch the approach you've taken? > > > On 17 Mar 2010, at 11:39, Jay K wrote: > > I have something working on Solaris now. > More details after testing on Linux and Darwin. > > - Jay > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Exec what? > You'd have to change the code to carefully reach the same place. > > - Jay > > Subject: Re: [M3devel] fork/cvsup > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 09:28:14 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not just exec in the child? > > On 17 Mar 2010, at 03:47, Jay K wrote: > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such time as one of the exec functions is > called. All APIs, including global data symbols, in any framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 18 00:04:23 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 23:04:23 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: <5EBE34A1-B22E-4E03-AB59-5A7E4ABFAAA9@cs.purdue.edu> References: , , , , ,,, , , ,,, , ,,, , ,,<553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, ,,, , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, , , , <5EBE34A1-B22E-4E03-AB59-5A7E4ABFAAA9@cs.purdue.edu> Message-ID: [somewhat self follow up] These ideas are not mutually exclusive. It should be easier to switch to user threads. One might imagine "IMPORT UserThread AS Thread" though with that construct, you'd have a system using a mix of each thread type, depending on what a module imported, on a module by module basis. Anyway, let me try something that doesn't involve user threads. Something like what I sent but smaller. I also need to see if user threads exhibit the "only it works 9 times" problem, and if I can debug that either way. It really seems to be a fixed number of times. Hm. I have been prone to say -C 99. I should make sure that isn't interpreted as -C 9. I do sometimes say -C 2. And that's supposed to be sequential clients and I've seen what it does when it hits the limit it prints a specific message (some forms do hit the limit, because the hangs are in child servers and the master server knows they are still running). Hm. I'll have to dig in more/again. I'm still not comfortable with my use of SuspendOthers. Is it guaranteed where such threads are? I know they are sitting in the signal header, on some platforms, but no guarantee what got interrupted by it, right? They don't have any of the Thread.m3 locks? But they might have some other locks? I guess like the atfork documentation says, for this stuff to work, everyone with a lock has to use atfork. Good point about the server, though it never historically has. We're stuck between user threads used to work for it and kernel threads probably never have. Or it could be my theories are all wrong. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 18:40:17 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup That makes little sense. Surely the cvsup server would want to run with proper multi-threading? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 18:18, Jay K wrote: Maybe an easier approach is to offer a way in quake to select user threads, that then "sticks" automatically through runtime? That either implies build_standalone, or provides a differently named m3core.so? Replacing m3core.so doesn't work, and wouldn't solve this problem. Presumably one would want cvsup to coexist with non-user threads programs. You have expressed not liking these ideas in the past though. It wouldn't necessarily cost even a function pointer indirection for pthreads. That is, it wouldn't necessarily be an @M3userthreads runtime choice. The choice would be made at compile time and the appropriate functions called directly. Well, granted, the easiest way to implement that is "directly" through other functions. Programs that need "fork all" semantics could use user threads. We could alter the types slightly I think so they have the same fingerprints. So that everything wouldn't have to be recompiled, so that it'd be a build time replacement of one .lib or the other. Ie. first fix it so that swapping .so files works. And then make user_threads imply build_standalone. Does that make sense? To repeat: first adjust the two Thread.T to match, if that is possible. Not their code, just the types. And then build m3core twice. And have user threads imply build_standalone. It'd probably just be a matter of adding a few unused fields. I think I'll first try the "limited atfork" idea. That code is in hand already at least to start. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 22:05:25 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Tony, I don't know. Here is some "argument', but I'm not sure. Adding threads does something different. Such threads would share mutation to global state. I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:30:47 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 14:13, Jay K wrote: --- bad news: It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. Restart manually. Works again. Around 9 times. Then hangs again. That is on Linux/x86 and Solaris/sparc. Doesn't work at all on Mac/amd64, just hangs. --- sketch: m3core uses pthread_atfork to selectively reinitialize Mainly to only have one thread. common Thread.PThreadAtFork is provided for others to do the same It is deliberately in a portable interface. Thread.ReforkThreadAfterProcessFork Is provided for users to restart threads from their child AtFork hander. This is used by the allocator/collector. Thread.ForkProcessAndAllThreads() Is used by "lazy" clients who want to restart all their threads but didn't keep track of them. The runtime can do it for them. This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads or not, depending on if they need their threads restarted. The runtime takes care of its threads either way. --- What'd I'd written up: attached works typically 9 times on Linux and Solaris before server hangs again. No improvement on Darwin, just hangs. Can't see much in debuggers for some reason. There is extra allowance in the m3core change such that users of fork + do work (as opposed to fork + exec) may or may not call ForkAll, depending on if they feel a need for their own threads to be recreated, and if they've kept track of how to recreate them, or just rely on the runtime to know all the threads. There are three runtime threads that are sometimes created in the parent, and if so, recreated in the child. background collector, foreground collector, weak ref thread I'll try to poke at it some more. I'm not sure what is the best way to suspend all threads. I tried a few differnt ways. SuspendOthers LockHeap pthread_mutex_lock various combinations It is deliberate that pthread specific code is in common/Thread.i3. That way code can be portable, at least among the two Posix thread implementations. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:31 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 18 00:12:09 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 17 Mar 2010 23:12:09 +0000 Subject: [M3devel] fork/cvsup In-Reply-To: <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> References: , , , ,,, , ,,, ,,, , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, , <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> Message-ID: I don't know. I know there is one thread that it merely shuts right down in the child. It does that by queueing a message to it. The initial problem you see in the current code is it hangs trying to do that. You can basically just remove that code (except then it won't work with user threads probably!). The problem you hit after that is ThreadPThread.m3 getting, I forget the errno, but pthread_kill complains that you give it a nonexistant thread. That's presumably because the child process has inherited the parent's data as to existant threads. "limited atfork" addresses that. "limited atfork" means, a subset of the diff I sent. So the next things for me to try: - verify user threads doesn't fail after 9 also - verify that 9 isn't associated with "-C 99". - assuming no to both of those, try "limited atfork" and remove the code to shutdown the (nonexistant) dispatcher thread. If that works, almost done. Only remaining part would be to expose a boolean from Thread.i3 so cvsup could make the right choice. There might be a way to structure the cvsup code to work either way and not have to know. Something like signaling the thread ahead of time that it might be going away, and unsignaling only in the parent. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 18:32:44 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup What are the expectations in the cvsup child regarding the threads it inherits? On 17 Mar 2010, at 18:05, Jay K wrote: Tony, I don't know. Here is some "argument', but I'm not sure. Adding threads does something different. Such threads would share mutation to global state. I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:30:47 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 17 Mar 2010, at 14:13, Jay K wrote: --- bad news: It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. Restart manually. Works again. Around 9 times. Then hangs again. That is on Linux/x86 and Solaris/sparc. Doesn't work at all on Mac/amd64, just hangs. --- sketch: m3core uses pthread_atfork to selectively reinitialize Mainly to only have one thread. common Thread.PThreadAtFork is provided for others to do the same It is deliberately in a portable interface. Thread.ReforkThreadAfterProcessFork Is provided for users to restart threads from their child AtFork hander. This is used by the allocator/collector. Thread.ForkProcessAndAllThreads() Is used by "lazy" clients who want to restart all their threads but didn't keep track of them. The runtime can do it for them. This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads or not, depending on if they need their threads restarted. The runtime takes care of its threads either way. --- What'd I'd written up: attached works typically 9 times on Linux and Solaris before server hangs again. No improvement on Darwin, just hangs. Can't see much in debuggers for some reason. There is extra allowance in the m3core change such that users of fork + do work (as opposed to fork + exec) may or may not call ForkAll, depending on if they feel a need for their own threads to be recreated, and if they've kept track of how to recreate them, or just rely on the runtime to know all the threads. There are three runtime threads that are sometimes created in the parent, and if so, recreated in the child. background collector, foreground collector, weak ref thread I'll try to poke at it some more. I'm not sure what is the best way to suspend all threads. I tried a few differnt ways. SuspendOthers LockHeap pthread_mutex_lock various combinations It is deliberate that pthread specific code is in common/Thread.i3. That way code can be portable, at least among the two Posix thread implementations. - Jay From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:31 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Can you sketch the approach you've taken? On 17 Mar 2010, at 11:39, Jay K wrote: I have something working on Solaris now. More details after testing on Linux and Darwin. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 14:01:15 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fork/cvsup Exec what? You'd have to change the code to carefully reach the same place. - Jay Subject: Re: [M3devel] fork/cvsup From: hosking at cs.purdue.edu Date: Wed, 17 Mar 2010 09:28:14 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not just exec in the child? On 17 Mar 2010, at 03:47, Jay K wrote: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. self. http://www.opengroup.org/onlinepubs/000095399/functions/fork.html Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. I've run through a few theories so far. Current thinking is related to what Tony said: use pthread_atfork: in prepare, stopworld in parent, resumeworld You don't want the child to be mid-gc for example, on another thread. Or mid-anything. in child, reinitialize -- current thread is the only thread Also in the cvsup code, ShutDown should just call DoShutDown immediately. I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. I don't have a theory as to why user threads work. I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). I was expecting more cases to act like mmap(shared), but none did, only it. I experimented with having mutexes and condition variables be initialize up front instead of on-demand. Via changing cvsup to lock/unlock or broadcast immediately upon creating them. On the theory that might let them work across process. That didn't make a difference. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Wed Mar 17 13:29:43 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Wed, 17 Mar 2010 13:29:43 +0100 Subject: [M3devel] cvsup In-Reply-To: References: , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> Message-ID: <1268828983.2906.0.camel@faramir.m3w.org> Yes, I can. I am building it regularly, from version to version, at least few times a year. And it also worked with my ancient kernel thread implementation. Was one of stress tests. On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > (Can anyone claim to have seen cvsup server work with kernel threads?) -- Dragi?a Duri? From jay.krell at cornell.edu Thu Mar 18 11:09:38 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 18 Mar 2010 10:09:38 +0000 Subject: [M3devel] cvsup In-Reply-To: <1268828983.2906.0.camel@faramir.m3w.org> References: ,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , <1268828983.2906.0.camel@faramir.m3w.org> Message-ID: Dragisha, Really? The server? With the -C flag and/or -f? Evidence is very very very very very good as to what is going on here. It is all related to "fork1" vs. "forkall". fork1 is the overwhelming usual behavior (Solaris has an option), and basically *any* library that uses pthreads is obligated to use pthread_atfork, be it a C library or the Modula-3 library, etc.. The application cannot do things, unless the library exposes its internal locks. The Posix spec for pthread_atfork explains the problem. Consider C code that links in Modula-3 code. C code is fairly free to use the fork + do work model. It isn't very common, but it does exist, and people have gone to extra length to keep it viable. bash and ccache I believe both use this model. Granted, if you had your own kernel thread implementation, it might have addressed this. I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions. The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork. I want to still try out the "bigger" change, but with the simpler lock/unlock. And test on Solaris and Darwin. I think for SuspendOthers to not work is not surprising. The threads can still hold a few locks even if suspended, I'm pretty sure. The point is to fork in a "controlled" fashion -- all locks held, and in the right order, so that both parent and child can release them. Taking "all" the locks in Prepare is more reliable. I do wonder if really "all" locks need to be taken. I only take the static ones declared in PThreadThread.c. - Jay > Subject: Re: [M3devel] cvsup > From: dragisha at m3w.org > To: jay.krell at cornell.edu > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > Yes, I can. I am building it regularly, from version to version, at > least few times a year. And it also worked with my ancient kernel thread > implementation. Was one of stress tests. > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Mar 18 13:56:11 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 18 Mar 2010 13:56:11 +0100 Subject: [M3devel] cvsup In-Reply-To: References: ,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> , ,<1268828983.2906.0.camel@faramir.m3w.org> Message-ID: <1268916971.2586.3.camel@faramir.m3w.org> Client, it's what I am using. And it's dead as we speak. Last worked before I pulled/built RC4 On Thu, 2010-03-18 at 10:09 +0000, Jay K wrote: > Dragisha, Really? The server? With the -C flag and/or -f? > > Evidence is very very very very very good as to what is going on here. > It is all related to "fork1" vs. "forkall". > fork1 is the overwhelming usual behavior (Solaris has an option), > and basically *any* library that uses pthreads is obligated to use > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > The application cannot do things, unless > the library exposes its internal locks. The Posix spec for > pthread_atfork explains the problem. > > Consider C code that links in Modula-3 code. > C code is fairly free to use the fork + do work model. It isn't very > common, > but it does exist, and people have gone to extra length to keep it > viable. > bash and ccache I believe both use this model. > > > Granted, if you had your own kernel thread implementation, it might > have addressed this. > > > I have a version now that has served over 1000 requests, similar to > what I sent, but a bit reduced. The main change was I just called > pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead > of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just > Linux. This version also doesn't expose the ForkProcessAndAllThreads > functions. > The client has to restart its threads, or in the cvsupd case, don't > depend on the dispatcher thread to survive fork. > I want to still try out the "bigger" change, but with the simpler > lock/unlock. > And test on Solaris and Darwin. > > > I think for SuspendOthers to not work is not surprising. > The threads can still hold a few locks even if suspended, I'm pretty > sure. > The point is to fork in a "controlled" fashion -- all locks held, and > in > the right order, so that both parent and child can release them. > > > Taking "all" the locks in Prepare is more reliable. > > > I do wonder if really "all" locks need to be taken. > I only take the static ones declared in PThreadThread.c. > > - Jay > > > > Subject: Re: [M3devel] cvsup > > From: dragisha at m3w.org > > To: jay.krell at cornell.edu > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > Yes, I can. I am building it regularly, from version to version, at > > least few times a year. And it also worked with my ancient kernel > thread > > implementation. Was one of stress tests. > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > (Can anyone claim to have seen cvsup server work with kernel > threads?) > > -- > > Dragi?a Duri? > > -- Dragi?a Duri? From wagner at elegosoft.com Thu Mar 18 15:25:10 2010 From: wagner at elegosoft.com (Olaf Wagner) Date: Thu, 18 Mar 2010 15:25:10 +0100 Subject: [M3devel] fork/cvsup In-Reply-To: References: , , , ,,, , ,,, ,,, , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, , <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> Message-ID: <20100318152510.nr94m5wi1wkw0048@mail.elegosoft.com> I'm not able to follow your discussion completely now, but some information about CVSup can be found at http://www.cvsup.org/ http://www.cvsup.org/howsofast.html AFAIK it uses the traditional Unix daemon pattern to fork a new server process for each client connection, which then creates threads for the streaming protocol. It should be possible to change the pattern to use threads only, but I'd rather like it if we could support the traditional Unix daemon pattern in a general form, too. Modula-3 is a systems programming language after all. Olaf Quoting Jay K : > > I don't know. I know there is one thread that it merely shuts right > down in the child. > > It does that by queueing a message to it. > > The initial problem you see in the current code is it hangs trying > to do that. > > > > > > You can basically just remove that code (except then it won't work > > with user threads probably!). > > > > > > The problem you hit after that is ThreadPThread.m3 getting, > > I forget the errno, but pthread_kill complains that you give > > it a nonexistant thread. That's presumably because the child > > process has inherited the parent's data as to existant threads. > > "limited atfork" addresses that. "limited atfork" means, a subset > > of the diff I sent. > > > > > > So the next things for me to try: > > - verify user threads doesn't fail after 9 also > > - verify that 9 isn't associated with "-C 99". > > - assuming no to both of those, try "limited atfork" and > > remove the code to shutdown the (nonexistant) dispatcher > > thread. If that works, almost done. Only remaining part would > > be to expose a boolean from Thread.i3 so cvsup could > > make the right choice. There might be a way to structure > > the cvsup code to work either way and not have to know. > > Something like signaling the thread ahead of time that it might > > be going away, and unsignaling only in the parent. > > > > > > - Jay > > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 18:32:44 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > What are the expectations in the cvsup child regarding the threads > it inherits? > > > > > On 17 Mar 2010, at 18:05, Jay K wrote: > > Tony, I don't know. > Here is some "argument', but I'm not sure. > > > Adding threads does something different. Such threads would share > mutation to global state. > I'm not a big fan of this model, but fork lets you establish some > perhaps expensive to establish state, then share it cheaply among a > bunch of future threads/processes, that may make their own local > modifications to it. One would have to read the cvsup code a bunch > to determine what it actually does and requires. > > I do suspect there is a general solution. Leaving anyone who uses > platform specific functions to fend for themselves seems a bit > unfair. Which functions to we abtract away in m3ore vs. which do we > leave > people to use on their own? And does that list change much in time? > Well, infinity isn't possible either, granted. And we've only seen > one program so far that cares, we shouldn't spend too much just for > one program. > > > There may be a smaller related fix, where m3core internally uses > atfork, but doesn't expose ForkAll to the client. I know cvsup has > the dispatcher thread that it expects to be inherited by children, > however all it does with it is queue a request to it to shut itself > down. In that way, ForkAll is a waste -- it recreates a thread, only > so the client can shut it down. I can pursue that more. > > > - Jay > > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:30:47 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > I don't think there is a "general" solution to this that should be > applied to the thread library. Modula-3 does not mandate any > support for fork! It is not part of the language. If a program > relies on platform-specific interfaces then it must be the one to > handle situations arising from the problem. Why does cvsup need to > fork in the first place? Surely it can simply add threads to handle > clients as they arrive? > > > > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > On 17 Mar 2010, at 14:13, Jay K wrote: > > --- > bad news: > It doesn't completely work. It works a bunch of times in a row, like > 9, then hangs. > Restart manually. Works again. Around 9 times. Then hangs again. > That is on Linux/x86 and Solaris/sparc. > Doesn't work at all on Mac/amd64, just hangs. > > --- > sketch: > m3core uses pthread_atfork to selectively reinitialize > Mainly to only have one thread. > > > common Thread.PThreadAtFork is provided for others to do the same > It is deliberately in a portable interface. > > > Thread.ReforkThreadAfterProcessFork > Is provided for users to restart threads from their child AtFork hander. > This is used by the allocator/collector. > > > Thread.ForkProcessAndAllThreads() > Is used by "lazy" clients who want to restart all their threads > but didn't keep track of them. The runtime can do it for them. > > > This allows for "fork + do work" folks do call or not call > ForkProcessAndAllThreads > or not, depending on if they need their threads restarted. > The runtime takes care of its threads either way. > > > --- > What'd I'd written up: > > attached works typically 9 times on Linux and Solaris > before server hangs again. > > > No improvement on Darwin, just hangs. > Can't see much in debuggers for some reason. > > > There is extra allowance in the m3core change such > that users of fork + do work (as opposed to fork + exec) > may or may not call ForkAll, depending on if they > feel a need for their own threads to be recreated, > and if they've kept track of how to recreate them, > or just rely on the runtime to know all the threads. > > > There are three runtime threads that are sometimes > created in the parent, and if so, recreated in the child. > background collector, foreground collector, weak ref thread > > > I'll try to poke at it some more. > > > I'm not sure what is the best way to suspend all threads. > I tried a few differnt ways. > SuspendOthers > LockHeap > pthread_mutex_lock > various combinations > > > It is deliberate that pthread specific code is in common/Thread.i3. > That way code can be portable, at least among the two Posix thread > implementations. > > > - Jay > > > > > > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:31 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Can you sketch the approach you've taken? > > > > > On 17 Mar 2010, at 11:39, Jay K wrote: > > I have something working on Solaris now. > More details after testing on Linux and Darwin. > > - Jay > > > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 14:01:15 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fork/cvsup > > Exec what? > You'd have to change the code to carefully reach the same place. > > - Jay > > > > Subject: Re: [M3devel] fork/cvsup > From: hosking at cs.purdue.edu > Date: Wed, 17 Mar 2010 09:28:14 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > > > > Why not just exec in the child? > > > On 17 Mar 2010, at 03:47, Jay K wrote: > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html > > > There are limits to what you can do in the child process. To be > totally safe you should restrict your-self yourself > self to only executing async-signal safe operations until such > time as one of the exec functions is > called. All APIs, including global data symbols, in any > framework or library should be assumed to be > unsafe after a fork() unless explicitly documented to be safe > or async-signal safe. If you need to use > these frameworks in the child process, you must exec. In this > situation it is reasonable to exec your-self. yourself. > self. > > > http://www.opengroup.org/onlinepubs/000095399/functions/fork.html > > Consequently, to avoid errors, the child process may only execute > async-signal-safe operations until such time as one of theexec > functions is called. [THR] Fork handlers may be established by > means of the pthread_atfork() function in order to maintain > application invariants across fork() calls. > > > I've run through a few theories so far. > Current thinking is related to what Tony said: > use pthread_atfork: > in prepare, stopworld > in parent, resumeworld > You don't want the child to be mid-gc for example, on another > thread. Or mid-anything. > in child, reinitialize -- current thread is the only thread > > > Also in the cvsup code, ShutDown should just call DoShutDown immediately. > I did that, without m3core changes, and it hits an error in the > pthread code, signaling a nonexistant thread. > pthread_atfork/child should address that -- child shouldn't retain a > record of all the threads in the parent. > > > I don't have a theory as to why user threads work. > > > I experimented with malloc vs. static alloc vs. sbrk vs. > mmap(private) vs. mmap(shared). > I was expecting more cases to act like mmap(shared), but none did, only it. > > > I experimented with having mutexes and condition variables be > initialize up front instead of on-demand. > Via changing cvsup to lock/unlock or broadcast immediately upon > creating them. > On the theory that might let them work across process. > That didn't make a difference. > > > - Jay > > > > > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From hosking at cs.purdue.edu Thu Mar 18 15:31:15 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 18 Mar 2010 10:31:15 -0400 Subject: [M3devel] fork/cvsup In-Reply-To: <20100318152510.nr94m5wi1wkw0048@mail.elegosoft.com> References: , , , , , , , , , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, , <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> <20100318152510.nr94m5wi1wkw0048@mail.elegosoft.com> Message-ID: <318D6C6F-80BE-41F5-80CC-03387905439B@cs.purdue.edu> Agreed. I do seem to recall cvsupd working with system threads way back when I was testing things. On 18 Mar 2010, at 10:25, Olaf Wagner wrote: > I'm not able to follow your discussion completely now, but some > information about CVSup can be found at > > http://www.cvsup.org/ > http://www.cvsup.org/howsofast.html > > AFAIK it uses the traditional Unix daemon pattern to fork a new > server process for each client connection, which then creates threads > for the streaming protocol. > > It should be possible to change the pattern to use threads only, > but I'd rather like it if we could support the traditional Unix > daemon pattern in a general form, too. Modula-3 is a systems > programming language after all. > > Olaf > > Quoting Jay K : > >> >> I don't know. I know there is one thread that it merely shuts right down in the child. >> >> It does that by queueing a message to it. >> >> The initial problem you see in the current code is it hangs trying to do that. >> >> >> >> >> >> You can basically just remove that code (except then it won't work >> >> with user threads probably!). >> >> >> >> >> >> The problem you hit after that is ThreadPThread.m3 getting, >> >> I forget the errno, but pthread_kill complains that you give >> >> it a nonexistant thread. That's presumably because the child >> >> process has inherited the parent's data as to existant threads. >> >> "limited atfork" addresses that. "limited atfork" means, a subset >> >> of the diff I sent. >> >> >> >> >> >> So the next things for me to try: >> >> - verify user threads doesn't fail after 9 also >> >> - verify that 9 isn't associated with "-C 99". >> >> - assuming no to both of those, try "limited atfork" and >> >> remove the code to shutdown the (nonexistant) dispatcher >> >> thread. If that works, almost done. Only remaining part would >> >> be to expose a boolean from Thread.i3 so cvsup could >> >> make the right choice. There might be a way to structure >> >> the cvsup code to work either way and not have to know. >> >> Something like signaling the thread ahead of time that it might >> >> be going away, and unsignaling only in the parent. >> >> >> >> >> >> - Jay >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 18:32:44 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> What are the expectations in the cvsup child regarding the threads it inherits? >> >> >> >> >> On 17 Mar 2010, at 18:05, Jay K wrote: >> >> Tony, I don't know. >> Here is some "argument', but I'm not sure. >> >> >> Adding threads does something different. Such threads would share mutation to global state. >> I'm not a big fan of this model, but fork lets you establish some perhaps expensive to establish state, then share it cheaply among a bunch of future threads/processes, that may make their own local modifications to it. One would have to read the cvsup code a bunch to determine what it actually does and requires. >> >> I do suspect there is a general solution. Leaving anyone who uses platform specific functions to fend for themselves seems a bit unfair. Which functions to we abtract away in m3ore vs. which do we leave >> people to use on their own? And does that list change much in time? Well, infinity isn't possible either, granted. And we've only seen one program so far that cares, we shouldn't spend too much just for one program. >> >> >> There may be a smaller related fix, where m3core internally uses atfork, but doesn't expose ForkAll to the client. I know cvsup has the dispatcher thread that it expects to be inherited by children, however all it does with it is queue a request to it to shut itself down. In that way, ForkAll is a waste -- it recreates a thread, only so the client can shut it down. I can pursue that more. >> >> >> - Jay >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:30:47 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> I don't think there is a "general" solution to this that should be applied to the thread library. Modula-3 does not mandate any support for fork! It is not part of the language. If a program relies on platform-specific interfaces then it must be the one to handle situations arising from the problem. Why does cvsup need to fork in the first place? Surely it can simply add threads to handle clients as they arrive? >> >> >> >> >> Antony Hosking | Associate Professor | Computer Science | Purdue University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> On 17 Mar 2010, at 14:13, Jay K wrote: >> >> --- >> bad news: >> It doesn't completely work. It works a bunch of times in a row, like 9, then hangs. >> Restart manually. Works again. Around 9 times. Then hangs again. >> That is on Linux/x86 and Solaris/sparc. >> Doesn't work at all on Mac/amd64, just hangs. >> >> --- >> sketch: >> m3core uses pthread_atfork to selectively reinitialize >> Mainly to only have one thread. >> >> >> common Thread.PThreadAtFork is provided for others to do the same >> It is deliberately in a portable interface. >> >> >> Thread.ReforkThreadAfterProcessFork >> Is provided for users to restart threads from their child AtFork hander. >> This is used by the allocator/collector. >> >> >> Thread.ForkProcessAndAllThreads() >> Is used by "lazy" clients who want to restart all their threads >> but didn't keep track of them. The runtime can do it for them. >> >> >> This allows for "fork + do work" folks do call or not call ForkProcessAndAllThreads >> or not, depending on if they need their threads restarted. >> The runtime takes care of its threads either way. >> >> >> --- >> What'd I'd written up: >> >> attached works typically 9 times on Linux and Solaris >> before server hangs again. >> >> >> No improvement on Darwin, just hangs. >> Can't see much in debuggers for some reason. >> >> >> There is extra allowance in the m3core change such >> that users of fork + do work (as opposed to fork + exec) >> may or may not call ForkAll, depending on if they >> feel a need for their own threads to be recreated, >> and if they've kept track of how to recreate them, >> or just rely on the runtime to know all the threads. >> >> >> There are three runtime threads that are sometimes >> created in the parent, and if so, recreated in the child. >> background collector, foreground collector, weak ref thread >> >> >> I'll try to poke at it some more. >> >> >> I'm not sure what is the best way to suspend all threads. >> I tried a few differnt ways. >> SuspendOthers >> LockHeap >> pthread_mutex_lock >> various combinations >> >> >> It is deliberate that pthread specific code is in common/Thread.i3. >> That way code can be portable, at least among the two Posix thread implementations. >> >> >> - Jay >> >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:01:31 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> Can you sketch the approach you've taken? >> >> >> >> >> On 17 Mar 2010, at 11:39, Jay K wrote: >> >> I have something working on Solaris now. >> More details after testing on Linux and Darwin. >> >> - Jay >> >> >> >> From: jay.krell at cornell.edu >> To: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:01:15 +0000 >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> Exec what? >> You'd have to change the code to carefully reach the same place. >> >> - Jay >> >> >> >> Subject: Re: [M3devel] fork/cvsup >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 09:28:14 -0400 >> CC: m3devel at elegosoft.com >> To: jay.krell at cornell.edu >> >> >> >> >> Why not just exec in the child? >> >> >> On 17 Mar 2010, at 03:47, Jay K wrote: >> >> http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html >> >> >> There are limits to what you can do in the child process. To be totally safe you should restrict your-self yourself >> self to only executing async-signal safe operations until such time as one of the exec functions is >> called. All APIs, including global data symbols, in any framework or library should be assumed to be >> unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use >> these frameworks in the child process, you must exec. In this situation it is reasonable to exec your-self. yourself. >> self. >> >> >> http://www.opengroup.org/onlinepubs/000095399/functions/fork.html >> >> Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of theexec functions is called. [THR] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. >> >> >> I've run through a few theories so far. >> Current thinking is related to what Tony said: >> use pthread_atfork: >> in prepare, stopworld >> in parent, resumeworld >> You don't want the child to be mid-gc for example, on another thread. Or mid-anything. >> in child, reinitialize -- current thread is the only thread >> >> >> Also in the cvsup code, ShutDown should just call DoShutDown immediately. >> I did that, without m3core changes, and it hits an error in the pthread code, signaling a nonexistant thread. >> pthread_atfork/child should address that -- child shouldn't retain a record of all the threads in the parent. >> >> >> I don't have a theory as to why user threads work. >> >> >> I experimented with malloc vs. static alloc vs. sbrk vs. mmap(private) vs. mmap(shared). >> I was expecting more cases to act like mmap(shared), but none did, only it. >> >> >> I experimented with having mutexes and condition variables be initialize up front instead of on-demand. >> Via changing cvsup to lock/unlock or broadcast immediately upon creating them. >> On the theory that might let them work across process. >> That didn't make a difference. >> >> >> - Jay >> >> >> >> >> > > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > From jay.krell at cornell.edu Thu Mar 18 16:45:11 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 18 Mar 2010 15:45:11 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <1268828983.2906.0.camel@faramir.m3w.org>, Message-ID: To repeat myself: > and basically *any* library that uses pthreads is obligated to use > pthread_atfork, be it a C library or the Modula-3 library, etc.. This is the crux of the matter. We are being "bad pthread citizens" by not using pthread_atfork. Granted, we are probably in large company. There is a smaller fix and a larger fix, but it seems very clear we should take one of them. The small fix is: common/Thread.i3: add PThreadAtFork It does nothing on posix/nt. It calls pthread_atfork on pthread. In RTCollector.m3 give a child handler that stores FALSE in the three booleans that indicate the three threads have started. No prepare or parent handler I believe. Though I should check for global locks there. Probably the locks in ThreadPThread suffice? In ThreadPThread.m3 provide three handlers: prepare: lock "everything", at least the 5 static locks. I might try walking all the threads and locking them too. parent: unlock everything child: unlock everything and reinitialize the globals Note that the atfork handlers run in many more programs than cvsup. They would be used also with fork + exec. Perhaps a way to disable that -- no way in Posix, we could just provide a boolean to ourselves. The larger fix would be to provide a common/Thread.i3 function to call fork() *and* recreate all Modula-3 threads in the child process. That is what I sent out. In the second case, you also then need to allow that the caller may or may not use that function, so you still need atfork handlers, but they have to be slightly different, as the diffs I sent show. I believe strongly this is the way to go. It is how one is a "good pthread citizen". Now, granted, I don't think most libraries are. Witness the caveat about fork() in the Darwin manpage and I think even the Posix spec. But pthread_atfork is meant to solve this. I don't suggest a full audit and add atfork handlers everywhere. But m3core we can at least do, and that should satisfy cvsup. Should check libm3 too. Plus, as long as each module takes reponsibility for itself, it shouldn't be so hard. That is, I don't think the case is all that strong for providing the "forkall" function that is in the diffs I sent. I'll do more testing and send out diffs "later". It could be as long as a week, I have a bunch of things to do. Or maybe just a day. :) I tested a form of this fix + cvsup + user threads and it appears that cvsup can do one small thing and thereby work either way, without knowledge as to the way. That is, it can shutdown the dispatcher "directly", instead of queuing to it. I'm not even sure the dispatcher thread is really needed. As I said, I don't believe the application can handle this all itself. Any library that uses pthreads is obligated to play into the general mechanism. Generally the internals are not exposed for the application to do it. Treating Modula-3 as a closed system in which nobody uses anything outside of or "below" m3core/libm3 I don't think is practical. "applications", arguably defines as "processes", are often composed of fairly disparate bodies of code that don't necessarily expose much to each other. For example, they might each create their own worker threads and not expose them. This is what pthread_atfork is for. Actually Tony, you gave me the lead here. :) There is a problem that m3posixthreads (user) and pthreads semantics diverage, and..I haven't been able to think of an way to fix that. Well, it is actually easy to make "forkall" the default and only behavior actually, on user and pthreads (but not NT). But I don't see a way to make "fork1" the behavior for user threads. You can associate a pid with each thread, and notice when the pid changes, but I don't know which one thread you would keep, and you wouldn't necessarily be able to register pthread_atfork handlers, unless maybe user threads were used only on platforms that actually have pthreads.. And still there's no good way to it on NT I expect. - Jay From: jay.krell at cornell.edu To: dragisha at m3w.org Date: Thu, 18 Mar 2010 10:09:38 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup Dragisha, Really? The server? With the -C flag and/or -f? Evidence is very very very very very good as to what is going on here. It is all related to "fork1" vs. "forkall". fork1 is the overwhelming usual behavior (Solaris has an option), and basically *any* library that uses pthreads is obligated to use pthread_atfork, be it a C library or the Modula-3 library, etc.. The application cannot do things, unless the library exposes its internal locks. The Posix spec for pthread_atfork explains the problem. Consider C code that links in Modula-3 code. C code is fairly free to use the fork + do work model. It isn't very common, but it does exist, and people have gone to extra length to keep it viable. bash and ccache I believe both use this model. Granted, if you had your own kernel thread implementation, it might have addressed this. I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions. The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork. I want to still try out the "bigger" change, but with the simpler lock/unlock. And test on Solaris and Darwin. I think for SuspendOthers to not work is not surprising. The threads can still hold a few locks even if suspended, I'm pretty sure. The point is to fork in a "controlled" fashion -- all locks held, and in the right order, so that both parent and child can release them. Taking "all" the locks in Prepare is more reliable. I do wonder if really "all" locks need to be taken. I only take the static ones declared in PThreadThread.c. - Jay > Subject: Re: [M3devel] cvsup > From: dragisha at m3w.org > To: jay.krell at cornell.edu > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > Yes, I can. I am building it regularly, from version to version, at > least few times a year. And it also worked with my ancient kernel thread > implementation. Was one of stress tests. > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdp at polstra.com Thu Mar 18 17:56:57 2010 From: jdp at polstra.com (John Polstra) Date: Thu, 18 Mar 2010 09:56:57 -0700 Subject: [M3devel] fork/cvsup In-Reply-To: <20100318152510.nr94m5wi1wkw0048@mail.elegosoft.com> References: , , , , , , , , , , , , , , , <553AF450-8285-41C4-9B8C-BC60AA0CAC5E@cs.purdue.edu>, , , , <39A5881D-E585-4674-99AB-90C087AD3319@cs.purdue.edu>, , <4FE1711F-313E-499A-85E0-80B9B3E99318@cs.purdue.edu> <20100318152510.nr94m5wi1wkw0048@mail.elegosoft.com> Message-ID: <79503C57-BC05-44E5-B0E2-494639A1FBD7@polstra.com> I don't want to get too involved in this, because it's been years and years since I even glanced at CVSup. (I retired from the software business a couple years ago.) But yes, it forks a new process per client and uses threads for the streaming protocol within each client process. I tried doing it all with threads when I first wrote it, but that turned out to be a bad idea for reasons of robustness. With the all-threads approach, any internal error (assert failure, bounds check, etc.) for any client will kill *all* clients. That's unacceptable from the user's point of view. You are on the right track with this bug. There is some kind of bad interaction between the forks and the threads system. I remember some issues along the same lines when I was developing the software, and I had to find what worked by experiment. At that time there were no standard facilities for using fork within threaded programs. Now that you have switched to using OS-provided threads (a good change, I think), it's not surprising that some problems have cropped up. John On Mar 18, 2010, at 7:25 AM, Olaf Wagner wrote: > I'm not able to follow your discussion completely now, but some > information about CVSup can be found at > > http://www.cvsup.org/ > http://www.cvsup.org/howsofast.html > > AFAIK it uses the traditional Unix daemon pattern to fork a new > server process for each client connection, which then creates threads > for the streaming protocol. > > It should be possible to change the pattern to use threads only, > but I'd rather like it if we could support the traditional Unix > daemon pattern in a general form, too. Modula-3 is a systems > programming language after all. > > Olaf > > Quoting Jay K : > >> >> I don't know. I know there is one thread that it merely shuts >> right down in the child. >> >> It does that by queueing a message to it. >> >> The initial problem you see in the current code is it hangs trying >> to do that. >> >> >> >> >> >> You can basically just remove that code (except then it won't work >> >> with user threads probably!). >> >> >> >> >> >> The problem you hit after that is ThreadPThread.m3 getting, >> >> I forget the errno, but pthread_kill complains that you give >> >> it a nonexistant thread. That's presumably because the child >> >> process has inherited the parent's data as to existant threads. >> >> "limited atfork" addresses that. "limited atfork" means, a subset >> >> of the diff I sent. >> >> >> >> >> >> So the next things for me to try: >> >> - verify user threads doesn't fail after 9 also >> >> - verify that 9 isn't associated with "-C 99". >> >> - assuming no to both of those, try "limited atfork" and >> >> remove the code to shutdown the (nonexistant) dispatcher >> >> thread. If that works, almost done. Only remaining part would >> >> be to expose a boolean from Thread.i3 so cvsup could >> >> make the right choice. There might be a way to structure >> >> the cvsup code to work either way and not have to know. >> >> Something like signaling the thread ahead of time that it might >> >> be going away, and unsignaling only in the parent. >> >> >> >> >> >> - Jay >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 18:32:44 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> What are the expectations in the cvsup child regarding the threads >> it inherits? >> >> >> >> >> On 17 Mar 2010, at 18:05, Jay K wrote: >> >> Tony, I don't know. >> Here is some "argument', but I'm not sure. >> >> >> Adding threads does something different. Such threads would share >> mutation to global state. >> I'm not a big fan of this model, but fork lets you establish some >> perhaps expensive to establish state, then share it cheaply among >> a bunch of future threads/processes, that may make their own >> local modifications to it. One would have to read the cvsup code a >> bunch to determine what it actually does and requires. >> >> I do suspect there is a general solution. Leaving anyone who uses >> platform specific functions to fend for themselves seems a bit >> unfair. Which functions to we abtract away in m3ore vs. which do >> we leave >> people to use on their own? And does that list change much in >> time? Well, infinity isn't possible either, granted. And we've >> only seen one program so far that cares, we shouldn't spend too >> much just for one program. >> >> >> There may be a smaller related fix, where m3core internally uses >> atfork, but doesn't expose ForkAll to the client. I know cvsup has >> the dispatcher thread that it expects to be inherited by children, >> however all it does with it is queue a request to it to shut >> itself down. In that way, ForkAll is a waste -- it recreates a >> thread, only so the client can shut it down. I can pursue that more. >> >> >> - Jay >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:30:47 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> I don't think there is a "general" solution to this that should be >> applied to the thread library. Modula-3 does not mandate any >> support for fork! It is not part of the language. If a program >> relies on platform-specific interfaces then it must be the one to >> handle situations arising from the problem. Why does cvsup need >> to fork in the first place? Surely it can simply add threads to >> handle clients as they arrive? >> >> >> >> >> Antony Hosking | Associate Professor | Computer Science | Purdue >> University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> On 17 Mar 2010, at 14:13, Jay K wrote: >> >> --- >> bad news: >> It doesn't completely work. It works a bunch of times in a row, >> like 9, then hangs. >> Restart manually. Works again. Around 9 times. Then hangs again. >> That is on Linux/x86 and Solaris/sparc. >> Doesn't work at all on Mac/amd64, just hangs. >> >> --- >> sketch: >> m3core uses pthread_atfork to selectively reinitialize >> Mainly to only have one thread. >> >> >> common Thread.PThreadAtFork is provided for others to do the same >> It is deliberately in a portable interface. >> >> >> Thread.ReforkThreadAfterProcessFork >> Is provided for users to restart threads from their child AtFork >> hander. >> This is used by the allocator/collector. >> >> >> Thread.ForkProcessAndAllThreads() >> Is used by "lazy" clients who want to restart all their threads >> but didn't keep track of them. The runtime can do it for them. >> >> >> This allows for "fork + do work" folks do call or not call >> ForkProcessAndAllThreads >> or not, depending on if they need their threads restarted. >> The runtime takes care of its threads either way. >> >> >> --- >> What'd I'd written up: >> >> attached works typically 9 times on Linux and Solaris >> before server hangs again. >> >> >> No improvement on Darwin, just hangs. >> Can't see much in debuggers for some reason. >> >> >> There is extra allowance in the m3core change such >> that users of fork + do work (as opposed to fork + exec) >> may or may not call ForkAll, depending on if they >> feel a need for their own threads to be recreated, >> and if they've kept track of how to recreate them, >> or just rely on the runtime to know all the threads. >> >> >> There are three runtime threads that are sometimes >> created in the parent, and if so, recreated in the child. >> background collector, foreground collector, weak ref thread >> >> >> I'll try to poke at it some more. >> >> >> I'm not sure what is the best way to suspend all threads. >> I tried a few differnt ways. >> SuspendOthers >> LockHeap >> pthread_mutex_lock >> various combinations >> >> >> It is deliberate that pthread specific code is in common/Thread.i3. >> That way code can be portable, at least among the two Posix thread >> implementations. >> >> >> - Jay >> >> >> >> >> >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:01:31 -0400 >> To: jay.krell at cornell.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> Can you sketch the approach you've taken? >> >> >> >> >> On 17 Mar 2010, at 11:39, Jay K wrote: >> >> I have something working on Solaris now. >> More details after testing on Linux and Darwin. >> >> - Jay >> >> >> >> From: jay.krell at cornell.edu >> To: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 14:01:15 +0000 >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] fork/cvsup >> >> Exec what? >> You'd have to change the code to carefully reach the same place. >> >> - Jay >> >> >> >> Subject: Re: [M3devel] fork/cvsup >> From: hosking at cs.purdue.edu >> Date: Wed, 17 Mar 2010 09:28:14 -0400 >> CC: m3devel at elegosoft.com >> To: jay.krell at cornell.edu >> >> >> >> >> Why not just exec in the child? >> >> >> On 17 Mar 2010, at 03:47, Jay K wrote: >> >> http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fork.2.html >> >> >> There are limits to what you can do in the child process. To be >> totally safe you should restrict your-self yourself >> self to only executing async-signal safe operations until such >> time as one of the exec functions is >> called. All APIs, including global data symbols, in any >> framework or library should be assumed to be >> unsafe after a fork() unless explicitly documented to be safe >> or async-signal safe. If you need to use >> these frameworks in the child process, you must exec. In this >> situation it is reasonable to exec your-self. yourself. >> self. >> >> >> http://www.opengroup.org/onlinepubs/000095399/functions/fork.html >> >> Consequently, to avoid errors, the child process may only execute >> async-signal-safe operations until such time as one of theexec >> functions is called. [THR] Fork handlers may be established by >> means of the pthread_atfork() function in order to maintain >> application invariants across fork() calls. >> >> >> I've run through a few theories so far. >> Current thinking is related to what Tony said: >> use pthread_atfork: >> in prepare, stopworld >> in parent, resumeworld >> You don't want the child to be mid-gc for example, on another >> thread. Or mid-anything. >> in child, reinitialize -- current thread is the only thread >> >> >> Also in the cvsup code, ShutDown should just call DoShutDown >> immediately. >> I did that, without m3core changes, and it hits an error in the >> pthread code, signaling a nonexistant thread. >> pthread_atfork/child should address that -- child shouldn't retain >> a record of all the threads in the parent. >> >> >> I don't have a theory as to why user threads work. >> >> >> I experimented with malloc vs. static alloc vs. sbrk vs. >> mmap(private) vs. mmap(shared). >> I was expecting more cases to act like mmap(shared), but none did, >> only it. >> >> >> I experimented with having mutexes and condition variables be >> initialize up front instead of on-demand. >> Via changing cvsup to lock/unlock or broadcast immediately upon >> creating them. >> On the theory that might let them work across process. >> That didn't make a difference. >> >> >> - Jay >> >> >> >> >> > > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, > Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 > 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: > Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: > DE163214194 > From hosking at cs.purdue.edu Thu Mar 18 18:16:03 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 18 Mar 2010 13:16:03 -0400 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <1268828983.2906.0.camel@faramir.m3w.org>, Message-ID: On 18 Mar 2010, at 11:45, Jay K wrote: > To repeat myself: > > > and basically *any* library that uses pthreads is obligated to use > > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > > This is the crux of the matter. > We are being "bad pthread citizens" by not using pthread_atfork. > Granted, we are probably in large company. > > > There is a smaller fix and a larger fix, but it seems very clear we > should take one of them. > > > The small fix is: > common/Thread.i3: add PThreadAtFork Name should probably be something more like Thread.AtFork. > It does nothing on posix/nt. > It calls pthread_atfork on pthread. > > > In RTCollector.m3 give a child handler that stores FALSE in the > three booleans that indicate the three threads have started. > No prepare or parent handler I believe. > Though I should check for global locks there. > Probably the locks in ThreadPThread suffice? I wonder if we should only fork when the collector is turned off? > In ThreadPThread.m3 provide three handlers: > prepare: lock "everything", at least the 5 static locks. > I might try walking all the threads and locking them too. > parent: unlock everything > child: unlock everything and reinitialize the globals > > > Note that the atfork handlers run in many more programs than cvsup. > They would be used also with fork + exec. fork + exec is probably relatively safe already. > Perhaps a way to disable that -- no way in Posix, we could just > provide a boolean to ourselves. Not sure I follow. > The larger fix would be to provide a common/Thread.i3 function > to call fork() *and* recreate all Modula-3 threads in the child process. > That is what I sent out. That's tough to do portably! > In the second case, you also then need to allow that the caller > may or may not use that function, so you still need atfork handlers, > but they have to be slightly different, as the diffs I sent show. > > > I believe strongly this is the way to go. > It is how one is a "good pthread citizen". > Now, granted, I don't think most libraries are. > Witness the caveat about fork() in the Darwin manpage and > I think even the Posix spec. > But pthread_atfork is meant to solve this. > > > I don't suggest a full audit and add atfork handlers everywhere. > But m3core we can at least do, and that should satisfy cvsup. > Should check libm3 too. Really we only need to be concerned with internal threads to the run-time, right? It's up to apps to restart threads in the children as necessary. > Plus, as long as each module takes reponsibility for itself, > it shouldn't be so hard. > That is, I don't think the case is all that strong for providing the "forkall" > function that is in the diffs I sent. Yes, I think that is overkill. > I'll do more testing and send out diffs "later". > It could be as long as a week, I have a bunch of things to do. > Or maybe just a day. :) > > > I tested a form of this fix + cvsup + user threads and it appears > that cvsup can do one small thing and thereby work either way, > without knowledge as to the way. > > > That is, it can shutdown the dispatcher "directly", instead of queuing to it. > I'm not even sure the dispatcher thread is really needed. > > > As I said, I don't believe the application can handle this all itself. > Any library that uses pthreads is obligated to play into the general > mechanism. Generally the internals are not exposed for the application > to do it. > > Treating Modula-3 as a closed system in which nobody uses > anything outside of or "below" m3core/libm3 I don't think is practical. > > > "applications", arguably defines as "processes", are often composed > of fairly disparate bodies of code that don't necessarily expose much > to each other. For example, they might each create their own worker > threads and not expose them. > This is what pthread_atfork is for. > Actually Tony, you gave me the lead here. :) ;-) > > > There is a problem that m3posixthreads (user) and pthreads semantics > diverage, and..I haven't been able to think of an way to fix that. > > > Well, it is actually easy to make "forkall" the default and only behavior actually, > on user and pthreads (but not NT). > > > But I don't see a way to make "fork1" the behavior for user threads. > You can associate a pid with each thread, and notice when the pid changes, > but I don't know which one thread you would keep, and you wouldn't > necessarily be able to register pthread_atfork handlers, unless maybe > user threads were used only on platforms that actually have pthreads.. > > > And still there's no good way to it on NT I expect. > > > - Jay > > From: jay.krell at cornell.edu > To: dragisha at m3w.org > Date: Thu, 18 Mar 2010 10:09:38 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > Dragisha, Really? The server? With the -C flag and/or -f? > > Evidence is very very very very very good as to what is going on here. > It is all related to "fork1" vs. "forkall". > fork1 is the overwhelming usual behavior (Solaris has an option), > and basically *any* library that uses pthreads is obligated to use > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > The application cannot do things, unless > the library exposes its internal locks. The Posix spec for > pthread_atfork explains the problem. > > Consider C code that links in Modula-3 code. > C code is fairly free to use the fork + do work model. It isn't very common, > but it does exist, and people have gone to extra length to keep it viable. > bash and ccache I believe both use this model. > > > Granted, if you had your own kernel thread implementation, it might > have addressed this. > > > I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions. > The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork. > I want to still try out the "bigger" change, but with the simpler lock/unlock. > And test on Solaris and Darwin. > > > I think for SuspendOthers to not work is not surprising. > The threads can still hold a few locks even if suspended, I'm pretty sure. > The point is to fork in a "controlled" fashion -- all locks held, and in > the right order, so that both parent and child can release them. > > > Taking "all" the locks in Prepare is more reliable. > > > I do wonder if really "all" locks need to be taken. > I only take the static ones declared in PThreadThread.c. > > - Jay > > > > Subject: Re: [M3devel] cvsup > > From: dragisha at m3w.org > > To: jay.krell at cornell.edu > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > Yes, I can. I am building it regularly, from version to version, at > > least few times a year. And it also worked with my ancient kernel thread > > implementation. Was one of stress tests. > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > -- > > Dragi?a Duri? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Mar 18 21:55:06 2010 From: jay.krell at cornell.edu (Jay K) Date: Thu, 18 Mar 2010 20:55:06 +0000 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <1268828983.2906.0.camel@faramir.m3w.org>, , Message-ID: > Name should probably be something more like Thread.AtFork. I disagree. It should be PThreadAtFork, because it only does anything when using pthreads. If you are on a non-pthread system (user threads or Win32), I expect, I think, the function won't have any affect. Granted, we could be using user threads on a system that has pthreads, so there is ambiguity, there is ambiguity either way. Does it mean, hey, thin wrapper over pthread_atfork, and I'll call it even if using user threads? Or does it mean, only call it if using pthreads? > I wonder if we should only fork when the collector is turned off? I don't quite follow. You mean, anyone calling fork() must GC.Disable()? Or LockHeap() Or, take all the thread locks? That last point is part of what I'm saying, since the "prepare" callback would do that. pthread_atfork lets you establish preconditions for fork() "distributed" out around the code, including with access to internals. Without having to "coordinate" with the caller of fork(). > fork + exec is probably relatively safe already. Right, but this change "unnecessarily" alters it. The "prepare" callback would still run. You can't discern fork + noexec vs. fork + exec They start the same. I think we are converging on agreement. Let me do some more tuning (stylistic, diff reduction, not perf) and testing. Diffs later, maybe tonight, maybe in a week, depending on life. > Perhaps a way to disable that -- no way in Posix, we could just > provide a boolean to ourselves. > Not sure I follow. I meant, something like, in m3core/libm3 where we have a fork+exec sequence, set a boolean somewhere to tell all of our "prepare" handlers not to waste their time doing anything. fork + exec need not run the prepare handlers. And I worry that all their lock taking might deadlock..but I suspect that would indicate a bug. ? > recreate the threads > That's tough to do portably! Easy with user threads -- automatically happens. Easy with pthreads -- systems with fork(). Hard/impossible on Win32. > Really we only need to be concerned with internal threads > to the run-time, right? It's up to apps to restart threads > in the children as necessary. I can agree with that. It is kind of just a lazy app that says "hey, please recreate all my threads, I didn't keep track of them, so I don't know what to restart, I know you kept track of all of them, so please do it for me". You should be able to "imagine" what my diff looks like. And maybe ok it without seeing it? Anyway, I definitely want to see Darwin not hang first. - Jay Subject: Re: [M3devel] cvsup From: hosking at cs.purdue.edu Date: Thu, 18 Mar 2010 13:16:03 -0400 CC: dragisha at m3w.org; m3devel at elegosoft.com To: jay.krell at cornell.edu On 18 Mar 2010, at 11:45, Jay K wrote: To repeat myself: > and basically *any* library that uses pthreads is obligated to use > pthread_atfork, be it a C library or the Modula-3 library, etc.. This is the crux of the matter. We are being "bad pthread citizens" by not using pthread_atfork. Granted, we are probably in large company. There is a smaller fix and a larger fix, but it seems very clear we should take one of them. The small fix is: common/Thread.i3: add PThreadAtFork Name should probably be something more like Thread.AtFork. It does nothing on posix/nt. It calls pthread_atfork on pthread. In RTCollector.m3 give a child handler that stores FALSE in the three booleans that indicate the three threads have started. No prepare or parent handler I believe. Though I should check for global locks there. Probably the locks in ThreadPThread suffice? I wonder if we should only fork when the collector is turned off? In ThreadPThread.m3 provide three handlers: prepare: lock "everything", at least the 5 static locks. I might try walking all the threads and locking them too. parent: unlock everything child: unlock everything and reinitialize the globals Note that the atfork handlers run in many more programs than cvsup. They would be used also with fork + exec. fork + exec is probably relatively safe already. Perhaps a way to disable that -- no way in Posix, we could just provide a boolean to ourselves. Not sure I follow. The larger fix would be to provide a common/Thread.i3 function to call fork() *and* recreate all Modula-3 threads in the child process. That is what I sent out. That's tough to do portably! In the second case, you also then need to allow that the caller may or may not use that function, so you still need atfork handlers, but they have to be slightly different, as the diffs I sent show. I believe strongly this is the way to go. It is how one is a "good pthread citizen". Now, granted, I don't think most libraries are. Witness the caveat about fork() in the Darwin manpage and I think even the Posix spec. But pthread_atfork is meant to solve this. I don't suggest a full audit and add atfork handlers everywhere. But m3core we can at least do, and that should satisfy cvsup. Should check libm3 too. Really we only need to be concerned with internal threads to the run-time, right? It's up to apps to restart threads in the children as necessary. Plus, as long as each module takes reponsibility for itself, it shouldn't be so hard. That is, I don't think the case is all that strong for providing the "forkall" function that is in the diffs I sent. Yes, I think that is overkill. I'll do more testing and send out diffs "later". It could be as long as a week, I have a bunch of things to do. Or maybe just a day. :) I tested a form of this fix + cvsup + user threads and it appears that cvsup can do one small thing and thereby work either way, without knowledge as to the way. That is, it can shutdown the dispatcher "directly", instead of queuing to it. I'm not even sure the dispatcher thread is really needed. As I said, I don't believe the application can handle this all itself. Any library that uses pthreads is obligated to play into the general mechanism. Generally the internals are not exposed for the application to do it. Treating Modula-3 as a closed system in which nobody uses anything outside of or "below" m3core/libm3 I don't think is practical. "applications", arguably defines as "processes", are often composed of fairly disparate bodies of code that don't necessarily expose much to each other. For example, they might each create their own worker threads and not expose them. This is what pthread_atfork is for. Actually Tony, you gave me the lead here. :) ;-) There is a problem that m3posixthreads (user) and pthreads semantics diverage, and..I haven't been able to think of an way to fix that. Well, it is actually easy to make "forkall" the default and only behavior actually, on user and pthreads (but not NT). But I don't see a way to make "fork1" the behavior for user threads. You can associate a pid with each thread, and notice when the pid changes, but I don't know which one thread you would keep, and you wouldn't necessarily be able to register pthread_atfork handlers, unless maybe user threads were used only on platforms that actually have pthreads.. And still there's no good way to it on NT I expect. - Jay From: jay.krell at cornell.edu To: dragisha at m3w.org Date: Thu, 18 Mar 2010 10:09:38 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup Dragisha, Really? The server? With the -C flag and/or -f? Evidence is very very very very very good as to what is going on here. It is all related to "fork1" vs. "forkall". fork1 is the overwhelming usual behavior (Solaris has an option), and basically *any* library that uses pthreads is obligated to use pthread_atfork, be it a C library or the Modula-3 library, etc.. The application cannot do things, unless the library exposes its internal locks. The Posix spec for pthread_atfork explains the problem. Consider C code that links in Modula-3 code. C code is fairly free to use the fork + do work model. It isn't very common, but it does exist, and people have gone to extra length to keep it viable. bash and ccache I believe both use this model. Granted, if you had your own kernel thread implementation, it might have addressed this. I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions. The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork. I want to still try out the "bigger" change, but with the simpler lock/unlock. And test on Solaris and Darwin. I think for SuspendOthers to not work is not surprising. The threads can still hold a few locks even if suspended, I'm pretty sure. The point is to fork in a "controlled" fashion -- all locks held, and in the right order, so that both parent and child can release them. Taking "all" the locks in Prepare is more reliable. I do wonder if really "all" locks need to be taken. I only take the static ones declared in PThreadThread.c. - Jay > Subject: Re: [M3devel] cvsup > From: dragisha at m3w.org > To: jay.krell at cornell.edu > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > Yes, I can. I am building it regularly, from version to version, at > least few times a year. And it also worked with my ancient kernel thread > implementation. Was one of stress tests. > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Mar 18 22:33:30 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 18 Mar 2010 17:33:30 -0400 Subject: [M3devel] cvsup In-Reply-To: References: , , , <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <1268828983.2906.0.camel@faramir.m3w.org>, , Message-ID: First off, AtFork probably does not belong in the Thread interface. That is part of the language definition and we shouldn't monkey with it. I guess I don't understand the specs of what you are proposing. I can't say that I can immediately imagine your diff. On 18 Mar 2010, at 16:55, Jay K wrote: > > Name should probably be something more like Thread.AtFork. > > > I disagree. > It should be PThreadAtFork, because it only does anything when > using pthreads. If you are on a non-pthread system (user threads or Win32), > I expect, I think, the function won't have any affect. > Granted, we could be using user threads on a system that has pthreads, > so there is ambiguity, there is ambiguity either way. > > > Does it mean, hey, thin wrapper over pthread_atfork, and I'll call it > even if using user threads? Or does it mean, only call it if > using pthreads? > > > > I wonder if we should only fork when the collector is turned off? > > > I don't quite follow. > You mean, anyone calling fork() must GC.Disable()? > Or LockHeap() > Or, take all the thread locks? > That last point is part of what I'm saying, since the "prepare" callback > would do that. > > > pthread_atfork lets you establish preconditions for fork() "distributed" > out around the code, including with access to internals. > Without having to "coordinate" with the caller of fork(). > > > > fork + exec is probably relatively safe already. > > > Right, but this change "unnecessarily" alters it. > The "prepare" callback would still run. > You can't discern fork + noexec vs. fork + exec > They start the same. > > > I think we are converging on agreement. > Let me do some more tuning (stylistic, diff reduction, not perf) and testing. > Diffs later, maybe tonight, maybe in a week, depending on life. > > > > Perhaps a way to disable that -- no way in Posix, we could just > > provide a boolean to ourselves. > > Not sure I follow. > > > I meant, something like, in m3core/libm3 where we have a fork+exec > sequence, set a boolean somewhere to tell all of our "prepare" handlers > not to waste their time doing anything. fork + exec need not > run the prepare handlers. > And I worry that all their lock taking might deadlock..but I suspect > that would indicate a bug. ? > > > > recreate the threads > > That's tough to do portably! > > > Easy with user threads -- automatically happens. > Easy with pthreads -- systems with fork(). > Hard/impossible on Win32. > > > > Really we only need to be concerned with internal threads > > to the run-time, right? It's up to apps to restart threads > > in the children as necessary. > > > I can agree with that. > It is kind of just a lazy app that says "hey, please recreate > all my threads, I didn't keep track of them, so I don't > know what to restart, I know you kept track of all of them, > so please do it for me". > > > You should be able to "imagine" what my diff looks like. > And maybe ok it without seeing it? > > > Anyway, I definitely want to see Darwin not hang first. > > > - Jay > > > > > > > > Subject: Re: [M3devel] cvsup > From: hosking at cs.purdue.edu > Date: Thu, 18 Mar 2010 13:16:03 -0400 > CC: dragisha at m3w.org; m3devel at elegosoft.com > To: jay.krell at cornell.edu > > On 18 Mar 2010, at 11:45, Jay K wrote: > > To repeat myself: > > > and basically *any* library that uses pthreads is obligated to use > > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > > This is the crux of the matter. > We are being "bad pthread citizens" by not using pthread_atfork. > Granted, we are probably in large company. > > > There is a smaller fix and a larger fix, but it seems very clear we > should take one of them. > > > The small fix is: > common/Thread.i3: add PThreadAtFork > > Name should probably be something more like Thread.AtFork. > > It does nothing on posix/nt. > It calls pthread_atfork on pthread. > > > In RTCollector.m3 give a child handler that stores FALSE in the > three booleans that indicate the three threads have started. > No prepare or parent handler I believe. > Though I should check for global locks there. > Probably the locks in ThreadPThread suffice? > > I wonder if we should only fork when the collector is turned off? > > In ThreadPThread.m3 provide three handlers: > prepare: lock "everything", at least the 5 static locks. > I might try walking all the threads and locking them too. > parent: unlock everything > child: unlock everything and reinitialize the globals > > > Note that the atfork handlers run in many more programs than cvsup. > They would be used also with fork + exec. > > fork + exec is probably relatively safe already. > > Perhaps a way to disable that -- no way in Posix, we could just > provide a boolean to ourselves. > > Not sure I follow. > > The larger fix would be to provide a common/Thread.i3 function > to call fork() *and* recreate all Modula-3 threads in the child process. > That is what I sent out. > > That's tough to do portably! > > In the second case, you also then need to allow that the caller > may or may not use that function, so you still need atfork handlers, > but they have to be slightly different, as the diffs I sent show. > > > I believe strongly this is the way to go. > It is how one is a "good pthread citizen". > Now, granted, I don't think most libraries are. > Witness the caveat about fork() in the Darwin manpage and > I think even the Posix spec. > But pthread_atfork is meant to solve this. > > > I don't suggest a full audit and add atfork handlers everywhere. > But m3core we can at least do, and that should satisfy cvsup. > Should check libm3 too. > > Really we only need to be concerned with internal threads to the run-time, right? It's up to apps to restart threads in the children as necessary. > > Plus, as long as each module takes reponsibility for itself, > it shouldn't be so hard. > That is, I don't think the case is all that strong for providing the "forkall" > function that is in the diffs I sent. > > Yes, I think that is overkill. > > I'll do more testing and send out diffs "later". > It could be as long as a week, I have a bunch of things to do. > Or maybe just a day. :) > > > I tested a form of this fix + cvsup + user threads and it appears > that cvsup can do one small thing and thereby work either way, > without knowledge as to the way. > > > That is, it can shutdown the dispatcher "directly", instead of queuing to it. > I'm not even sure the dispatcher thread is really needed. > > > As I said, I don't believe the application can handle this all itself. > Any library that uses pthreads is obligated to play into the general > mechanism. Generally the internals are not exposed for the application > to do it. > > Treating Modula-3 as a closed system in which nobody uses > anything outside of or "below" m3core/libm3 I don't think is practical. > > > "applications", arguably defines as "processes", are often composed > of fairly disparate bodies of code that don't necessarily expose much > to each other. For example, they might each create their own worker > threads and not expose them. > This is what pthread_atfork is for. > Actually Tony, you gave me the lead here. :) > > ;-) > > > > There is a problem that m3posixthreads (user) and pthreads semantics > diverage, and..I haven't been able to think of an way to fix that. > > > Well, it is actually easy to make "forkall" the default and only behavior actually, > on user and pthreads (but not NT). > > > But I don't see a way to make "fork1" the behavior for user threads. > You can associate a pid with each thread, and notice when the pid changes, > but I don't know which one thread you would keep, and you wouldn't > necessarily be able to register pthread_atfork handlers, unless maybe > user threads were used only on platforms that actually have pthreads.. > > > And still there's no good way to it on NT I expect. > > > - Jay > > From: jay.krell at cornell.edu > To: dragisha at m3w.org > Date: Thu, 18 Mar 2010 10:09:38 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup > > Dragisha, Really? The server? With the -C flag and/or -f? > > Evidence is very very very very very good as to what is going on here. > It is all related to "fork1" vs. "forkall". > fork1 is the overwhelming usual behavior (Solaris has an option), > and basically *any* library that uses pthreads is obligated to use > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > The application cannot do things, unless > the library exposes its internal locks. The Posix spec for > pthread_atfork explains the problem. > > Consider C code that links in Modula-3 code. > C code is fairly free to use the fork + do work model. It isn't very common, > but it does exist, and people have gone to extra length to keep it viable. > bash and ccache I believe both use this model. > > > Granted, if you had your own kernel thread implementation, it might > have addressed this. > > > I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions. > The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork. > I want to still try out the "bigger" change, but with the simpler lock/unlock. > And test on Solaris and Darwin. > > > I think for SuspendOthers to not work is not surprising. > The threads can still hold a few locks even if suspended, I'm pretty sure. > The point is to fork in a "controlled" fashion -- all locks held, and in > the right order, so that both parent and child can release them. > > > Taking "all" the locks in Prepare is more reliable. > > > I do wonder if really "all" locks need to be taken. > I only take the static ones declared in PThreadThread.c. > > - Jay > > > > Subject: Re: [M3devel] cvsup > > From: dragisha at m3w.org > > To: jay.krell at cornell.edu > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > Yes, I can. I am building it regularly, from version to version, at > > least few times a year. And it also worked with my ancient kernel thread > > implementation. Was one of stress tests. > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > (Can anyone claim to have seen cvsup server work with kernel threads?) > > -- > > Dragi?a Duri? > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 17:01:03 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements Message-ID: refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 18:21:13 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: Message-ID: diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: atfork2-cvsup.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: atfork2-m3core.txt URL: From jay.krell at cornell.edu Fri Mar 19 19:08:59 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 18:08:59 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , Message-ID: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 19 19:13:32 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 19 Mar 2010 14:13:32 -0400 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , Message-ID: Why not put it into RTProcess? Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > > could go in any of ThreadF, RTOS, RTProcess? > > > Or RTThread. Or almost anywhere. > > > Some typos in the diffs: > in a comment, ThreadF__ should be RTOS__ > function PThreadForkHandler named inconsistently, > should be PThreadAtForkHandler > > > Again, "PThread" appears in these names to indicate > their meaning is not portable. Their existance is, but not their meaning. > We have no way of saying "only call this function for pthreads", > instead as I understand, we provide a portable interface with > drastically varying semantics, such as "do something" vs. "do nothing". > > > Given that Init in ThreadPThread.m3 is private, we could probably > take the stackbase parameter from the caller instead. > > > The call to AtFork should probably follow InitWithStackbase, > in case it fails under low memory, the Die/assert more likely to "work". > > > I'm not completely happy making RTOS public. > So maybe ThreadF? > I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. > But for now they don't exist. > > > Maybe a new interface? ThreadPThread.i3, but is still > present on all platforms, so mostly portable can call it. > > > ThreadPThread.AtFork? > PThread.AtFork? That seems right. > > > - Jay > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 17:21:13 +0000 > Subject: Re: [M3devel] cvsup fix refinements > > diffs attached > > > For now I've removed the Get/SetNextForkWillExec. > I also think truly get/set is right, not inc/dec. > It was confusing enough to interpret and initialize > the value. And it is also not clear what the default > should be. The usual behavor is exec does follow fork. > The safer default if the handlers work ok is to assume > exec will not follow. It is a perf hint or a don't change > how things were hint? Hint to speed up fork+exec or hint > to avoid running the new code that might not work? > > > (For now I've removed the Get/SetNextForkWillExec.) > They'd have to be implemented three times, and > I had trouble defining them. > Obviously Win32/userthreads just return a > hardcoded true, but it is hard to explain > from the caller of Get's point of view. > > Maybe: > SetNextForkWillExec > GetNextForkWillExec > > and then: > PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? > > That is, > someone is who is calling fork and possibly exec, > can immediately tell what these functions mean. > > > But the implementer of an atfork handler has to > do quite a semantic translation I think. > I wrote it backwards the first time. That either > implies it is confusing, or I wasn't thinking. > > > If it really is a problem to run this stuff when exec > will follow, we should know quickly, as building cm3 > is an aggressive user of this code. > > > This version has worked repeatedly on Darwin. > I didn't test user threads but it is structured such > as to make that obviously work. > The cvsup changes are in pthreaad_atfork handlers, > so no change for userthreads. > > > I didn't test on others but confidence is quite high now. > > description of changes at least where they are hard to read: > > m3core: > Init split into Init and InitWithStackBase, > We have to provide the old stack base in the child fork handler > instead of estimating from the address of a local. I don't > know what stack is used, maybe the caller of fork? > i.e. we might have eaten a fair amount of stack and > there could be arbitrary references above the current stack. > > > WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner > > There is a resulting double lock in WeakRefFromRef, every time. > Could instead copy/paste more code around, i.e.: > EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > > > - Jay > > > > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 16:01:03 +0000 > Subject: [M3devel] cvsup fix refinements > > refinements: > > > new public functions in m3core: > > > TYPE PThreadForkHandler = PROCEDURE(); > > <* EXTERNAL RTOS__PThreadAtFork *> > PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; > > > (That's not a change yet.) > > > PROCEDURE SetNextForkWillExec(value: BOOLEAN); > > PROCEDURE GetNextForkWillExec(): BOOLEAN; > > These are only an optimization and should > perhaps be removed? They should be used > under LockHeap/UnlockHeap? which wasn't previously > public. This way, existing fork/exec paths > not affected, though maybe though might as well > ought to be? > > > could go in any of ThreadF, RTOS, RTProcess? > > > Should be called under RTOS.LockHeap? > Therefore I put in RTOS. > Also helps that RTOS is exported from *Thread.m3. > RTOS not previously public. > > > Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) > I implemented them that way: true incs, false decs. > > > Or don't bother with them at all? > > > Furthermore, in RTCollector, instead of claiming the threads > aren't started, if they were started, restart them. > This makes more sense for the weak cleaner to me, and > seems reasonable for the other two. > > > Diffs not yet available. > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 19:15:34 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 18:15:34 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , , Message-ID: RTProcess is fine. Unix is a little wierd because I don't think it should do anything if using userthreads. Unix implies a fairly thin wrapper? - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:13:32 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not put it into RTProcess?Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 19:47:46 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 18:47:46 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , , , , , , Message-ID: How about DoesForkExist, DoesForkInheritAllThreads? From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 18:15:34 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements RTProcess is fine. Unix is a little wierd because I don't think it should do anything if using userthreads. Unix implies a fairly thin wrapper? - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:13:32 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not put it into RTProcess?Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 19 19:47:00 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 19 Mar 2010 14:47:00 -0400 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , Message-ID: <62DD30EE-66EB-444C-84D0-0C1C0B56CD19@cs.purdue.edu> I think the atfork support should go into Unix.i3. It's not necessary for Windows, since Unix.fork is not available there. Also, we should make user and system threading behave the same way. So, for user threads we would destroy all the other threads in the child. This implies that we should have an m3core process fork mechanism. Anyone wanting to fork can then rely on the same behaviour. > Index: src/m3core.h > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/m3core.h,v > retrieving revision 1.8 > diff -u -r1.8 m3core.h > --- src/m3core.h 19 Mar 2010 17:07:58 -0000 1.8 > +++ src/m3core.h 19 Mar 2010 17:08:41 -0000 > @@ -404,6 +404,13 @@ > UINT32 __cdecl Uin__htonl(UINT32 x); > UINT16 __cdecl Uin__htons(UINT16 x); > > +typedef void (*PThreadForkHandler)(void); > + > +INTEGER > +RTOS__PThreadAtFork(PThreadForkHandler prep, > + PThreadForkHandler parent, > + PThreadForkHandler child); > + > #ifdef __cplusplus > } /* extern "C" */ > #endif > Index: src/runtime/common/RTCollector.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/runtime/common/RTCollector.m3,v > retrieving revision 1.88 > diff -u -r1.88 RTCollector.m3 > --- src/runtime/common/RTCollector.m3 15 Dec 2009 19:52:08 -0000 1.88 > +++ src/runtime/common/RTCollector.m3 19 Mar 2010 17:08:42 -0000 > @@ -2033,19 +2033,31 @@ > > VAR startedWeakCleaner := FALSE; > > -PROCEDURE WeakRefFromRef (r: REFANY; p: WeakRefCleanUpProc := NIL): WeakRef = > +PROCEDURE StartWeakCleaner () = > VAR > start := FALSE; > - result: WeakRef; > BEGIN > - <* ASSERT r # NIL *> > TRY > RTOS.LockHeap(); > - (* create a WeakCleaner thread the first time through *) > - IF p # NIL AND NOT startedWeakCleaner THEN > + IF NOT startedWeakCleaner THEN > start := TRUE; > startedWeakCleaner := TRUE; > END; > + FINALLY > + RTOS.UnlockHeap(); > + END; > + IF start THEN > + EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > + END; > + END StartWeakCleaner; > + > +PROCEDURE WeakRefFromRef (r: REFANY; p: WeakRefCleanUpProc := NIL): WeakRef = > + VAR > + result: WeakRef; > + BEGIN > + <* ASSERT r # NIL *> > + TRY > + RTOS.LockHeap(); > (* if necessary, expand weakTable *) > IF weakFree0 = -1 THEN ExpandWeakTable(); END; > IF p # NIL THEN > @@ -2075,9 +2087,8 @@ > FINALLY > RTOS.UnlockHeap(); > END; > - IF start THEN > - EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > - END; > + (* create a WeakCleaner thread the first time through *) > + StartWeakCleaner(); > RETURN result; > END WeakRefFromRef; I would leave WeakRefFromRef unchanged, and simply restart the thread in the child if startedWeakCleaner is TRUE: PROCEDURE AtForkChild() = BEGIN RTOS.LockHeap(); IF startedForeground THEN StartBackgroundCollection(); END; IF startedBackground THEN StartBackgroundCollection(); END; IF startedWeakCleaner THEN StarteWeakCleaner(); END: END AtForkChild; > > @@ -2771,8 +2782,24 @@ > > (*** INITIALIZATION ***) > > +PROCEDURE PThreadForkChild() = > + BEGIN > + IF startedForeground THEN > + StartBackgroundCollection(); > + END; > + IF startedBackground THEN > + StartBackgroundCollection(); > + END; > + IF startedWeakCleaner THEN > + StartWeakCleaner(); > + END; > + END PThreadForkChild; > + > PROCEDURE Init () = > + VAR r: INTEGER; > BEGIN > + r := RTOS.PThreadAtFork((*prepare*)NIL, (*parent*)NIL, PThreadForkChild); > + <* ASSERT r = 0 *> > IF RTParams.IsPresent("paranoidgc") THEN InstallSanityCheck(); END; > IF RTParams.IsPresent("nogc") THEN disableCount := 1; END; > IF RTParams.IsPresent("noincremental") THEN incremental := FALSE; END; > Index: src/runtime/common/RTOS.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/runtime/common/RTOS.i3,v > retrieving revision 1.10 > diff -u -r1.10 RTOS.i3 > --- src/runtime/common/RTOS.i3 8 Sep 2009 05:54:55 -0000 1.10 > +++ src/runtime/common/RTOS.i3 19 Mar 2010 17:08:42 -0000 > @@ -5,7 +5,7 @@ > (*| modified on Sun Feb 21 14:15:00 PST 1993 by jdd *) > (*| modified on Wed Jan 27 22:27:27 PST 1993 by mjordan *) > > -(* "RTOS" is a private interface that provides the low-level, > +(* "RTOS" is a semi-private interface that provides the low-level, > OS-specific memory allocation and shutdown routines. *) Leave this private! > > INTERFACE RTOS; > @@ -40,4 +40,13 @@ > (* Write the "n" bytes beginning at address "a" to the standard > error output file or console. *) > > +(* ThreadF__PThreadAtFork calls pthread_atfork > + * on pthreads platform, otherwise does nothing. > + * Return value is from pthread_atform, 0 for success, > + * always 0 with user threads and Win32 threads. > + *) > +TYPE PThreadForkHandler = PROCEDURE(); > +<* EXTERNAL RTOS__PThreadAtFork *> > +PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; > + Put the pthread_atfork wrapper into Unix.i3. > END RTOS. > Index: src/runtime/common/m3makefile > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/runtime/common/m3makefile,v > retrieving revision 1.27 > diff -u -r1.27 m3makefile > --- src/runtime/common/m3makefile 14 Dec 2009 08:09:48 -0000 1.27 > +++ src/runtime/common/m3makefile 19 Mar 2010 17:08:42 -0000 > @@ -58,7 +58,7 @@ > Interface ("RTSignal") > Interface ("RTStack") > Interface ("RTTypeSRC") > -interface ("RTOS") > +Interface ("RTOS") > > proc FileExists(a) is > return not stale (a, a) > Index: src/thread/POSIX/ThreadPosixC.c > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v > retrieving revision 1.32 > diff -u -r1.32 ThreadPosixC.c > --- src/thread/POSIX/ThreadPosixC.c 16 Dec 2009 12:21:36 -0000 1.32 > +++ src/thread/POSIX/ThreadPosixC.c 19 Mar 2010 17:08:42 -0000 > @@ -279,6 +279,14 @@ > #endif > } > > +INTEGER > +RTOS__PThreadAtFork(PThreadForkHandler prep, > + PThreadForkHandler parent, > + PThreadForkHandler child) > +{ > + return 0; > +} > + > #ifdef __cplusplus > } /* extern "C" */ > #endif > Index: src/thread/PTHREAD/ThreadPThread.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/PTHREAD/ThreadPThread.m3,v > retrieving revision 1.233 > diff -u -r1.233 ThreadPThread.m3 > --- src/thread/PTHREAD/ThreadPThread.m3 18 Mar 2010 11:12:10 -0000 1.233 > +++ src/thread/PTHREAD/ThreadPThread.m3 19 Mar 2010 17:08:42 -0000 > @@ -6,7 +6,7 @@ > Scheduler, SchedulerPosix, RTOS, RTHooks, ThreadPThread; > > IMPORT Cerrno, FloatMode, MutexRep, > - RTCollectorSRC, RTError, RTHeapRep, RTIO, RTParams, > + RTCollectorSRC, RTError, RTHeapRep, RTIO, RTOS, RTParams, > RTPerfTool, RTProcess, ThreadEvent, Time, > Unix, Utime, Word, Usched, > Uerror, Uexec; > @@ -1294,18 +1294,18 @@ > > (*-------------------------------------------------------- Initialization ---*) > > -PROCEDURE Init ()= > +PROCEDURE InitWithStackBase (stackbase: ADDRESS) = > VAR > self: T; > me: Activation; > BEGIN > - InitC(ADR(self)); > + InitC(stackbase); > > me := NEW(Activation, > mutex := pthread_mutex_new(), > cond := pthread_cond_new()); > InitActivations(me); > - me.stackbase := ADR(self); (* not quite accurate but hopefully ok *) > + me.stackbase := stackbase; > IF me.mutex = NIL OR me.cond = NIL THEN > Die(ThisLine(), "Thread initialization failed."); > END; > @@ -1322,8 +1322,48 @@ > IF RTParams.IsPresent("foregroundgc") THEN > RTCollectorSRC.StartForegroundCollection(); > END; > + END InitWithStackBase; > + > +PROCEDURE Init ()= > + VAR r: INTEGER; > + BEGIN > + r := RTOS.PThreadAtFork(PThreadAtForkPrepare, PThreadAtForkParent, PThreadAtForkChild); > + IF r # 0 THEN DieI(ThisLine(), r) END; > + InitWithStackBase(ADR(r)); (* not quite accurate but hopefully ok *) > END Init; > > +VAR locks := ARRAY [0..3] OF pthread_mutex_t{ > + activeMu, slotsMu, initMu, perfMu}; > + > +PROCEDURE PThreadAtForkPrepare() = > + VAR r: int; > + BEGIN > + joinMu.acquire(); > + LockHeap(); > + FOR i := FIRST(locks) TO LAST(locks) DO > + r := pthread_mutex_lock(locks[i]); > + IF r # 0 THEN DieI(ThisLine(), r) END; > + END; > + (* Walk activations and lock all threads? *) > + END PThreadAtForkPrepare; I think there may be deadlock issues here! Would it not be better simply to SuspendOthers() in prepare and ResumeOthers() in parent/child? You would still need to acquire slotsMu/initMu/perfMu before SuspendOthers, and release them after ResumeOthers(). > + > +PROCEDURE PThreadAtForkParent() = > + VAR r: int; > + BEGIN > + FOR i := LAST(locks) TO FIRST(locks) BY -1 DO > + r := pthread_mutex_unlock(locks[i]); > + IF r # 0 THEN DieI(ThisLine(), r) END; > + END; > + UnlockHeap(); > + joinMu.release(); > + END PThreadAtForkParent; > + > +PROCEDURE PThreadAtForkChild() = > + BEGIN > + PThreadAtForkParent(); > + InitWithStackBase(GetActivation().stackbase); > + END PThreadAtForkChild; > + > (*------------------------------------------------------------- collector ---*) > (* These procedures provide synchronization primitives for the allocator > and collector. *) > Index: src/thread/PTHREAD/ThreadPThreadC.c > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/PTHREAD/ThreadPThreadC.c,v > retrieving revision 1.123 > diff -u -r1.123 ThreadPThreadC.c > --- src/thread/PTHREAD/ThreadPThreadC.c 25 Feb 2010 08:31:36 -0000 1.123 > +++ src/thread/PTHREAD/ThreadPThreadC.c 19 Mar 2010 17:08:42 -0000 > @@ -414,6 +414,14 @@ > #endif > } > > +INTEGER > +RTOS__PThreadAtFork(PThreadForkHandler prep, > + PThreadForkHandler parent, > + PThreadForkHandler child) > +{ > + return pthread_atfork(prep, parent, child); > +} > + > #ifdef __cplusplus > } /* extern "C" */ > #endif > Index: src/thread/WIN32/ThreadWin32C.c > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/WIN32/ThreadWin32C.c,v > retrieving revision 1.64 > diff -u -r1.64 ThreadWin32C.c > --- src/thread/WIN32/ThreadWin32C.c 16 Jan 2010 12:44:56 -0000 1.64 > +++ src/thread/WIN32/ThreadWin32C.c 19 Mar 2010 17:08:42 -0000 > @@ -146,6 +146,15 @@ > p(bottom, __getReg(?)); /* bsp? bspstore? try numbers until we find it? */ > #endif > } > + > +/*-------------------------------------------------------------------------*/ > + > +INTEGER > +RTOS__PThreadAtFork(PThreadForkHandler prep, > + PThreadForkHandler parent, > + PThreadForkHandler child) > +{ > + return 0; > } > > /*-------------------------------------------------------------------------*/ On 19 Mar 2010, at 14:13, Tony Hosking wrote: > Why not put it into RTProcess? > Or into Unix? > I want to avoid confusion between Thread.Fork (fork a thread) and process fork. > > On 19 Mar 2010, at 14:08, Jay K wrote: > >> > could go in any of ThreadF, RTOS, RTProcess? >> >> >> Or RTThread. Or almost anywhere. >> >> >> Some typos in the diffs: >> in a comment, ThreadF__ should be RTOS__ >> function PThreadForkHandler named inconsistently, >> should be PThreadAtForkHandler >> >> >> Again, "PThread" appears in these names to indicate >> their meaning is not portable. Their existance is, but not their meaning. >> We have no way of saying "only call this function for pthreads", >> instead as I understand, we provide a portable interface with >> drastically varying semantics, such as "do something" vs. "do nothing". >> >> >> Given that Init in ThreadPThread.m3 is private, we could probably >> take the stackbase parameter from the caller instead. >> >> >> The call to AtFork should probably follow InitWithStackbase, >> in case it fails under low memory, the Die/assert more likely to "work". >> >> >> I'm not completely happy making RTOS public. >> So maybe ThreadF? >> I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. >> But for now they don't exist. >> >> >> Maybe a new interface? ThreadPThread.i3, but is still >> present on all platforms, so mostly portable can call it. >> >> >> ThreadPThread.AtFork? >> PThread.AtFork? That seems right. >> >> >> - Jay >> >> >> From: jay.krell at cornell.edu >> To: m3devel at elegosoft.com >> Date: Fri, 19 Mar 2010 17:21:13 +0000 >> Subject: Re: [M3devel] cvsup fix refinements >> >> diffs attached >> >> >> For now I've removed the Get/SetNextForkWillExec. >> I also think truly get/set is right, not inc/dec. >> It was confusing enough to interpret and initialize >> the value. And it is also not clear what the default >> should be. The usual behavor is exec does follow fork. >> The safer default if the handlers work ok is to assume >> exec will not follow. It is a perf hint or a don't change >> how things were hint? Hint to speed up fork+exec or hint >> to avoid running the new code that might not work? >> >> >> (For now I've removed the Get/SetNextForkWillExec.) >> They'd have to be implemented three times, and >> I had trouble defining them. >> Obviously Win32/userthreads just return a >> hardcoded true, but it is hard to explain >> from the caller of Get's point of view. >> >> Maybe: >> SetNextForkWillExec >> GetNextForkWillExec >> >> and then: >> PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? >> >> That is, >> someone is who is calling fork and possibly exec, >> can immediately tell what these functions mean. >> >> >> But the implementer of an atfork handler has to >> do quite a semantic translation I think. >> I wrote it backwards the first time. That either >> implies it is confusing, or I wasn't thinking. >> >> >> If it really is a problem to run this stuff when exec >> will follow, we should know quickly, as building cm3 >> is an aggressive user of this code. >> >> >> This version has worked repeatedly on Darwin. >> I didn't test user threads but it is structured such >> as to make that obviously work. >> The cvsup changes are in pthreaad_atfork handlers, >> so no change for userthreads. >> >> >> I didn't test on others but confidence is quite high now. >> >> description of changes at least where they are hard to read: >> >> m3core: >> Init split into Init and InitWithStackBase, >> We have to provide the old stack base in the child fork handler >> instead of estimating from the address of a local. I don't >> know what stack is used, maybe the caller of fork? >> i.e. we might have eaten a fair amount of stack and >> there could be arbitrary references above the current stack. >> >> >> WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner >> >> There is a resulting double lock in WeakRefFromRef, every time. >> Could instead copy/paste more code around, i.e.: >> EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); >> >> >> - Jay >> >> >> >> >> >> >> From: jay.krell at cornell.edu >> To: m3devel at elegosoft.com >> Date: Fri, 19 Mar 2010 16:01:03 +0000 >> Subject: [M3devel] cvsup fix refinements >> >> refinements: >> >> >> new public functions in m3core: >> >> >> TYPE PThreadForkHandler = PROCEDURE(); >> >> <* EXTERNAL RTOS__PThreadAtFork *> >> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; >> >> >> (That's not a change yet.) >> >> >> PROCEDURE SetNextForkWillExec(value: BOOLEAN); >> >> PROCEDURE GetNextForkWillExec(): BOOLEAN; >> >> These are only an optimization and should >> perhaps be removed? They should be used >> under LockHeap/UnlockHeap? which wasn't previously >> public. This way, existing fork/exec paths >> not affected, though maybe though might as well >> ought to be? >> >> >> could go in any of ThreadF, RTOS, RTProcess? >> >> >> Should be called under RTOS.LockHeap? >> Therefore I put in RTOS. >> Also helps that RTOS is exported from *Thread.m3. >> RTOS not previously public. >> >> >> Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) >> I implemented them that way: true incs, false decs. >> >> >> Or don't bother with them at all? >> >> >> Furthermore, in RTCollector, instead of claiming the threads >> aren't started, if they were started, restart them. >> This makes more sense for the weak cleaner to me, and >> seems reasonable for the other two. >> >> >> Diffs not yet available. >> >> >> - Jay > From hosking at cs.purdue.edu Fri Mar 19 19:49:14 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 19 Mar 2010 14:49:14 -0400 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , , Message-ID: <7654F263-E59A-4DC0-8225-0F5F3E5BB6C1@cs.purdue.edu> On 19 Mar 2010, at 14:15, Jay K wrote: > RTProcess is fine. > Unix is a little wierd because I don't think it should do anything if using userthreads. I think forking with user threads should have the same behaviour as with system threads: all threads except the forker will die. Another thing. What happens to the dead threads in the child when they get GC'd? Their finaliser will try to invoke CleanThread which might fail because the threads mutex/cond are in a bad state in the child. Is that possible? > Unix implies a fairly thin wrapper? > > - Jay > > > Subject: Re: [M3devel] cvsup fix refinements > From: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 14:13:32 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not put it into RTProcess? > Or into Unix? > I want to avoid confusion between Thread.Fork (fork a thread) and process fork. > > On 19 Mar 2010, at 14:08, Jay K wrote: > > > could go in any of ThreadF, RTOS, RTProcess? > > > Or RTThread. Or almost anywhere. > > > Some typos in the diffs: > in a comment, ThreadF__ should be RTOS__ > function PThreadForkHandler named inconsistently, > should be PThreadAtForkHandler > > > Again, "PThread" appears in these names to indicate > their meaning is not portable. Their existance is, but not their meaning. > We have no way of saying "only call this function for pthreads", > instead as I understand, we provide a portable interface with > drastically varying semantics, such as "do something" vs. "do nothing". > > > Given that Init in ThreadPThread.m3 is private, we could probably > take the stackbase parameter from the caller instead. > > > The call to AtFork should probably follow InitWithStackbase, > in case it fails under low memory, the Die/assert more likely to "work". > > > I'm not completely happy making RTOS public. > So maybe ThreadF? > I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. > But for now they don't exist. > > > Maybe a new interface? ThreadPThread.i3, but is still > present on all platforms, so mostly portable can call it. > > > ThreadPThread.AtFork? > PThread.AtFork? That seems right. > > > - Jay > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 17:21:13 +0000 > Subject: Re: [M3devel] cvsup fix refinements > > diffs attached > > > For now I've removed the Get/SetNextForkWillExec. > I also think truly get/set is right, not inc/dec. > It was confusing enough to interpret and initialize > the value. And it is also not clear what the default > should be. The usual behavor is exec does follow fork. > The safer default if the handlers work ok is to assume > exec will not follow. It is a perf hint or a don't change > how things were hint? Hint to speed up fork+exec or hint > to avoid running the new code that might not work? > > > (For now I've removed the Get/SetNextForkWillExec.) > They'd have to be implemented three times, and > I had trouble defining them. > Obviously Win32/userthreads just return a > hardcoded true, but it is hard to explain > from the caller of Get's point of view. > > Maybe: > SetNextForkWillExec > GetNextForkWillExec > > and then: > PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? > > That is, > someone is who is calling fork and possibly exec, > can immediately tell what these functions mean. > > > But the implementer of an atfork handler has to > do quite a semantic translation I think. > I wrote it backwards the first time. That either > implies it is confusing, or I wasn't thinking. > > > If it really is a problem to run this stuff when exec > will follow, we should know quickly, as building cm3 > is an aggressive user of this code. > > > This version has worked repeatedly on Darwin. > I didn't test user threads but it is structured such > as to make that obviously work. > The cvsup changes are in pthreaad_atfork handlers, > so no change for userthreads. > > > I didn't test on others but confidence is quite high now. > > description of changes at least where they are hard to read: > > m3core: > Init split into Init and InitWithStackBase, > We have to provide the old stack base in the child fork handler > instead of estimating from the address of a local. I don't > know what stack is used, maybe the caller of fork? > i.e. we might have eaten a fair amount of stack and > there could be arbitrary references above the current stack. > > > WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner > > There is a resulting double lock in WeakRefFromRef, every time. > Could instead copy/paste more code around, i.e.: > EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > > > - Jay > > > > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 16:01:03 +0000 > Subject: [M3devel] cvsup fix refinements > > refinements: > > > new public functions in m3core: > > > TYPE PThreadForkHandler = PROCEDURE(); > > <* EXTERNAL RTOS__PThreadAtFork *> > PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; > > > (That's not a change yet.) > > > PROCEDURE SetNextForkWillExec(value: BOOLEAN); > > PROCEDURE GetNextForkWillExec(): BOOLEAN; > > These are only an optimization and should > perhaps be removed? They should be used > under LockHeap/UnlockHeap? which wasn't previously > public. This way, existing fork/exec paths > not affected, though maybe though might as well > ought to be? > > > could go in any of ThreadF, RTOS, RTProcess? > > > Should be called under RTOS.LockHeap? > Therefore I put in RTOS. > Also helps that RTOS is exported from *Thread.m3. > RTOS not previously public. > > > Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) > I implemented them that way: true incs, false decs. > > > Or don't bother with them at all? > > > Furthermore, in RTCollector, instead of claiming the threads > aren't started, if they were started, restart them. > This makes more sense for the weak cleaner to me, and > seems reasonable for the other two. > > > Diffs not yet available. > > > - Jay > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 19 19:50:01 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 19 Mar 2010 14:50:01 -0400 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , , , , , , Message-ID: <83C24180-1290-4F1D-A777-D283475A543F@cs.purdue.edu> The way fork is defined on modern platforms I don't think the child should ever inherit all the threads. On 19 Mar 2010, at 14:47, Jay K wrote: > How about DoesForkExist, DoesForkInheritAllThreads? > > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 18:15:34 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup fix refinements > > RTProcess is fine. > Unix is a little wierd because I don't think it should do anything if using userthreads. > Unix implies a fairly thin wrapper? > > - Jay > > > Subject: Re: [M3devel] cvsup fix refinements > From: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 14:13:32 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not put it into RTProcess? > Or into Unix? > I want to avoid confusion between Thread.Fork (fork a thread) and process fork. > > On 19 Mar 2010, at 14:08, Jay K wrote: > > > could go in any of ThreadF, RTOS, RTProcess? > > > Or RTThread. Or almost anywhere. > > > Some typos in the diffs: > in a comment, ThreadF__ should be RTOS__ > function PThreadForkHandler named inconsistently, > should be PThreadAtForkHandler > > > Again, "PThread" appears in these names to indicate > their meaning is not portable. Their existance is, but not their meaning. > We have no way of saying "only call this function for pthreads", > instead as I understand, we provide a portable interface with > drastically varying semantics, such as "do something" vs. "do nothing". > > > Given that Init in ThreadPThread.m3 is private, we could probably > take the stackbase parameter from the caller instead. > > > The call to AtFork should probably follow InitWithStackbase, > in case it fails under low memory, the Die/assert more likely to "work". > > > I'm not completely happy making RTOS public. > So maybe ThreadF? > I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. > But for now they don't exist. > > > Maybe a new interface? ThreadPThread.i3, but is still > present on all platforms, so mostly portable can call it. > > > ThreadPThread.AtFork? > PThread.AtFork? That seems right. > > > - Jay > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 17:21:13 +0000 > Subject: Re: [M3devel] cvsup fix refinements > > diffs attached > > > For now I've removed the Get/SetNextForkWillExec. > I also think truly get/set is right, not inc/dec. > It was confusing enough to interpret and initialize > the value. And it is also not clear what the default > should be. The usual behavor is exec does follow fork. > The safer default if the handlers work ok is to assume > exec will not follow. It is a perf hint or a don't change > how things were hint? Hint to speed up fork+exec or hint > to avoid running the new code that might not work? > > > (For now I've removed the Get/SetNextForkWillExec.) > They'd have to be implemented three times, and > I had trouble defining them. > Obviously Win32/userthreads just return a > hardcoded true, but it is hard to explain > from the caller of Get's point of view. > > Maybe: > SetNextForkWillExec > GetNextForkWillExec > > and then: > PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? > > That is, > someone is who is calling fork and possibly exec, > can immediately tell what these functions mean. > > > But the implementer of an atfork handler has to > do quite a semantic translation I think. > I wrote it backwards the first time. That either > implies it is confusing, or I wasn't thinking. > > > If it really is a problem to run this stuff when exec > will follow, we should know quickly, as building cm3 > is an aggressive user of this code. > > > This version has worked repeatedly on Darwin. > I didn't test user threads but it is structured such > as to make that obviously work. > The cvsup changes are in pthreaad_atfork handlers, > so no change for userthreads. > > > I didn't test on others but confidence is quite high now. > > description of changes at least where they are hard to read: > > m3core: > Init split into Init and InitWithStackBase, > We have to provide the old stack base in the child fork handler > instead of estimating from the address of a local. I don't > know what stack is used, maybe the caller of fork? > i.e. we might have eaten a fair amount of stack and > there could be arbitrary references above the current stack. > > > WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner > > There is a resulting double lock in WeakRefFromRef, every time. > Could instead copy/paste more code around, i.e.: > EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > > > - Jay > > > > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 16:01:03 +0000 > Subject: [M3devel] cvsup fix refinements > > refinements: > > > new public functions in m3core: > > > TYPE PThreadForkHandler = PROCEDURE(); > > <* EXTERNAL RTOS__PThreadAtFork *> > PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; > > > (That's not a change yet.) > > > PROCEDURE SetNextForkWillExec(value: BOOLEAN); > > PROCEDURE GetNextForkWillExec(): BOOLEAN; > > These are only an optimization and should > perhaps be removed? They should be used > under LockHeap/UnlockHeap? which wasn't previously > public. This way, existing fork/exec paths > not affected, though maybe though might as well > ought to be? > > > could go in any of ThreadF, RTOS, RTProcess? > > > Should be called under RTOS.LockHeap? > Therefore I put in RTOS. > Also helps that RTOS is exported from *Thread.m3. > RTOS not previously public. > > > Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) > I implemented them that way: true incs, false decs. > > > Or don't bother with them at all? > > > Furthermore, in RTCollector, instead of claiming the threads > aren't started, if they were started, restart them. > This makes more sense for the weak cleaner to me, and > seems reasonable for the other two. > > > Diffs not yet available. > > > - Jay > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 20:56:04 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 19:56:04 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: <83C24180-1290-4F1D-A777-D283475A543F@cs.purdue.edu> References: , , , , , , , , , , , , <83C24180-1290-4F1D-A777-D283475A543F@cs.purdue.edu> Message-ID: Agreed. Details later. From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:50:01 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements The way fork is defined on modern platforms I don't think the child should ever inherit all the threads. On 19 Mar 2010, at 14:47, Jay K wrote:How about DoesForkExist, DoesForkInheritAllThreads? From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 18:15:34 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements RTProcess is fine. Unix is a little wierd because I don't think it should do anything if using userthreads. Unix implies a fairly thin wrapper? - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:13:32 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not put it into RTProcess?Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Mar 19 21:17:34 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 19 Mar 2010 16:17:34 -0400 Subject: [M3devel] cvsup fix refinements In-Reply-To: References: , , , , , , , , , , , , <83C24180-1290-4F1D-A777-D283475A543F@cs.purdue.edu> Message-ID: <70C491E7-6699-4556-A92C-763B22CBD6DA@cs.purdue.edu> I meant for both user and system threads. On 19 Mar 2010, at 15:56, Jay K wrote: > Agreed. Details later. > > From: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 14:50:01 -0400 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup fix refinements > > The way fork is defined on modern platforms I don't think the child should ever inherit all the threads. > > On 19 Mar 2010, at 14:47, Jay K wrote: > > How about DoesForkExist, DoesForkInheritAllThreads? > > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 18:15:34 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] cvsup fix refinements > > RTProcess is fine. > Unix is a little wierd because I don't think it should do anything if using userthreads. > Unix implies a fairly thin wrapper? > > - Jay > > > Subject: Re: [M3devel] cvsup fix refinements > From: hosking at cs.purdue.edu > Date: Fri, 19 Mar 2010 14:13:32 -0400 > CC: m3devel at elegosoft.com > To: jay.krell at cornell.edu > > Why not put it into RTProcess? > Or into Unix? > I want to avoid confusion between Thread.Fork (fork a thread) and process fork. > > On 19 Mar 2010, at 14:08, Jay K wrote: > > > could go in any of ThreadF, RTOS, RTProcess? > > > Or RTThread. Or almost anywhere. > > > Some typos in the diffs: > in a comment, ThreadF__ should be RTOS__ > function PThreadForkHandler named inconsistently, > should be PThreadAtForkHandler > > > Again, "PThread" appears in these names to indicate > their meaning is not portable. Their existance is, but not their meaning. > We have no way of saying "only call this function for pthreads", > instead as I understand, we provide a portable interface with > drastically varying semantics, such as "do something" vs. "do nothing". > > > Given that Init in ThreadPThread.m3 is private, we could probably > take the stackbase parameter from the caller instead. > > > The call to AtFork should probably follow InitWithStackbase, > in case it fails under low memory, the Die/assert more likely to "work". > > > I'm not completely happy making RTOS public. > So maybe ThreadF? > I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. > But for now they don't exist. > > > Maybe a new interface? ThreadPThread.i3, but is still > present on all platforms, so mostly portable can call it. > > > ThreadPThread.AtFork? > PThread.AtFork? That seems right. > > > - Jay > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 17:21:13 +0000 > Subject: Re: [M3devel] cvsup fix refinements > > diffs attached > > > For now I've removed the Get/SetNextForkWillExec. > I also think truly get/set is right, not inc/dec. > It was confusing enough to interpret and initialize > the value. And it is also not clear what the default > should be. The usual behavor is exec does follow fork. > The safer default if the handlers work ok is to assume > exec will not follow. It is a perf hint or a don't change > how things were hint? Hint to speed up fork+exec or hint > to avoid running the new code that might not work? > > > (For now I've removed the Get/SetNextForkWillExec.) > They'd have to be implemented three times, and > I had trouble defining them. > Obviously Win32/userthreads just return a > hardcoded true, but it is hard to explain > from the caller of Get's point of view. > > Maybe: > SetNextForkWillExec > GetNextForkWillExec > > and then: > PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? > > That is, > someone is who is calling fork and possibly exec, > can immediately tell what these functions mean. > > > But the implementer of an atfork handler has to > do quite a semantic translation I think. > I wrote it backwards the first time. That either > implies it is confusing, or I wasn't thinking. > > > If it really is a problem to run this stuff when exec > will follow, we should know quickly, as building cm3 > is an aggressive user of this code. > > > This version has worked repeatedly on Darwin. > I didn't test user threads but it is structured such > as to make that obviously work. > The cvsup changes are in pthreaad_atfork handlers, > so no change for userthreads. > > > I didn't test on others but confidence is quite high now. > > description of changes at least where they are hard to read: > > m3core: > Init split into Init and InitWithStackBase, > We have to provide the old stack base in the child fork handler > instead of estimating from the address of a local. I don't > know what stack is used, maybe the caller of fork? > i.e. we might have eaten a fair amount of stack and > there could be arbitrary references above the current stack. > > > WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner > > There is a resulting double lock in WeakRefFromRef, every time. > Could instead copy/paste more code around, i.e.: > EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); > > > - Jay > > > > > > > From: jay.krell at cornell.edu > To: m3devel at elegosoft.com > Date: Fri, 19 Mar 2010 16:01:03 +0000 > Subject: [M3devel] cvsup fix refinements > > refinements: > > > new public functions in m3core: > > > TYPE PThreadForkHandler = PROCEDURE(); > > <* EXTERNAL RTOS__PThreadAtFork *> > PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; > > > (That's not a change yet.) > > > PROCEDURE SetNextForkWillExec(value: BOOLEAN); > > PROCEDURE GetNextForkWillExec(): BOOLEAN; > > These are only an optimization and should > perhaps be removed? They should be used > under LockHeap/UnlockHeap? which wasn't previously > public. This way, existing fork/exec paths > not affected, though maybe though might as well > ought to be? > > > could go in any of ThreadF, RTOS, RTProcess? > > > Should be called under RTOS.LockHeap? > Therefore I put in RTOS. > Also helps that RTOS is exported from *Thread.m3. > RTOS not previously public. > > > Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) > I implemented them that way: true incs, false decs. > > > Or don't bother with them at all? > > > Furthermore, in RTCollector, instead of claiming the threads > aren't started, if they were started, restart them. > This makes more sense for the weak cleaner to me, and > seems reasonable for the other two. > > > Diffs not yet available. > > > - Jay > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 23:49:15 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 22:49:15 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: <70C491E7-6699-4556-A92C-763B22CBD6DA@cs.purdue.edu> References: , , , , , , , , , , , , <83C24180-1290-4F1D-A777-D283475A543F@cs.purdue.edu> , <70C491E7-6699-4556-A92C-763B22CBD6DA@cs.purdue.edu> Message-ID: I was going to ask: Do you mean pthreads or Modula-3 threads? How you think it is or how you think it should be? Modula-3 user threads are inherited. That is what this function would say. At least they are currently. However, as you assert, I've also wondered many times to myself the past few days: Should Modula-3 userthreads and pthreads act the same way? Always? Optionally? What stopped me was I couldn't convince myself it is trivial to implement. I was thinking, like, you'd associate a pid with each thread. If getpid doesn't match thread.pid, the thread is from before a fork. I then worried that you'd build up all the data about threads "forever", across multiple forks. However that is probably false. As well, if you provide RTProcess.Fork() as the replacement for fork(), it affords more power or at least ease of implementation. I guess where you end is like: RTProcess.Fork() does nothing on Windows in fact is <* EXTERNAL *> and fails to link most likely You could have the convention of returning ENOSYS but I say no. Or it might be a NIL pointer, following the convention you know about, have used several times. just calls fork() on pthreads on user threads, described later RTProcess.AtFork() (maybe RegisterAtFork or RegisterForkCallbacks?) does nothing on Windows (but exists and links, usable by portable code) might be NIL on pthreads just calls pthread_atfork on user threads, described later And then RTProcess.Fork(), RTProcess.AtFork on user threads work together to provide basically the same meaning as fork() + pthread_fork() has on pthreads. Easy to imagine how to code it actually. AtFork maintains an array of function pointers. Fork() calls them and fork in the proscribed order. The array is inherited across processes (like pthread_atfork()). Only the calling thread survives, like fork(). I didn't realize it at first, but killing of all other threads is pretty easy. You just reinitialize like I did for pthreads. Then the next time the timer expires, it doesn't find them to be scheduled. I also strongly suspect that in libm3 where fork+exec is called, some of what it does -- in particular turning off the timer -- would be moved to RTProcess.Fork(). Or perhaps that is in ThreadPosix.m3's AtForkPrepare. Either way, the general idea is that fork is easier to use, be it fork + exec, or even fork + do work. You replace it with RTProcess.Fork(). As well however, fork + exec on pthreads continues to work. You don't have to call RTProcess.Fork(). However RTProcess.Fork() is needed to provide matching semantics in user threads if you don't exec after fork. Geez this is a mouthful. I think it is somewhat important to dwell on -- what semantics change, which external libraries just work and don't have to be wrapped. if you are doing fork + exec, you can keep doing the same, I think if you don't care about user threads, and want them all to be inherited, you can keep doing the same, I think No working code should have an altered meaning. cvsupd on pthreads doesn't count as working code, of course. In fact no fork + do work code on pthreads probably worked. My assertion that "we should use pthread_atfork and be good pthread citizens" remains true and respected by these changes. I'll code this up over the next few days/week. I'm not sure anything ends up in Unix, could all end up in RTProcessC.c or such. If RTProcess.i3 isn't public, it will be. Or we can introduce a new interface. This all sounds about right? Notice that, like my second set of diffs, existing pthreads fork + exec paths will be "significantly" affected -- taking all the locks in the prepare call. It should work. - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 16:17:34 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu I meant for both user and system threads. On 19 Mar 2010, at 15:56, Jay K wrote: Agreed. Details later. From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:50:01 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements The way fork is defined on modern platforms I don't think the child should ever inherit all the threads. On 19 Mar 2010, at 14:47, Jay K wrote: How about DoesForkExist, DoesForkInheritAllThreads? From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 18:15:34 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements RTProcess is fine. Unix is a little wierd because I don't think it should do anything if using userthreads. Unix implies a fairly thin wrapper? - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:13:32 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not put it into RTProcess? Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Mar 19 23:58:08 2010 From: jay.krell at cornell.edu (Jay K) Date: Fri, 19 Mar 2010 22:58:08 +0000 Subject: [M3devel] cvsup fix refinements In-Reply-To: <7654F263-E59A-4DC0-8225-0F5F3E5BB6C1@cs.purdue.edu> References: , , , , , , , <7654F263-E59A-4DC0-8225-0F5F3E5BB6C1@cs.purdue.edu> Message-ID: > What happens to the dead threads in the child when they get GC'd? > Their finaliser will try to invoke CleanThread which might fail because > the threads mutex/cond are in a bad state in the child. Is that possible? This is what pthread_atfork is for. I'm not sure you don't realize that, or are pointing out that I wasn't handling everything. You see the question mark in the ThreadPThread.AtForkPrepare in the diff I sent? That covers this..and I think indeed it needs work: I think for this reason Thread.AtForkPrepare should walk all the threads and take their locks, and then child will release them all, and we can go ahead and cleanup them up right away as well (in the child), we might otherwise lose all references to them, since they were in thread locals. I think AtForkPrepare needs to broadly speaking, acquire *all* locks. Not just some. The diffs I sent only take some. I will include this change in next set of diffs -- the ones that will also change user threads to not inherit threads after RTProcess.Fork(). I suspect this is a difficult task in some code bases. And I'm not sure I suggest clean this up throughout cm3. However we can at least address RTCollector.m3 and ThreadPThread.m3 and get cvsup working. And then we could be like Apple and the docs could just say "fork not followed by exec not guaranteed to work much". ? Also makes me wonder..we should provide wrapped up fork+exec, eh, we already do. It is in libm3. It should probably be in m3core, but I don't think that difference matters too much. - Jay From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:49:14 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] cvsup fix refinements On 19 Mar 2010, at 14:15, Jay K wrote: RTProcess is fine. Unix is a little wierd because I don't think it should do anything if using userthreads. I think forking with user threads should have the same behaviour as with system threads: all threads except the forker will die. Another thing. What happens to the dead threads in the child when they get GC'd? Their finaliser will try to invoke CleanThread which might fail because the threads mutex/cond are in a bad state in the child. Is that possible? Unix implies a fairly thin wrapper? - Jay Subject: Re: [M3devel] cvsup fix refinements From: hosking at cs.purdue.edu Date: Fri, 19 Mar 2010 14:13:32 -0400 CC: m3devel at elegosoft.com To: jay.krell at cornell.edu Why not put it into RTProcess? Or into Unix? I want to avoid confusion between Thread.Fork (fork a thread) and process fork. On 19 Mar 2010, at 14:08, Jay K wrote: > could go in any of ThreadF, RTOS, RTProcess? Or RTThread. Or almost anywhere. Some typos in the diffs: in a comment, ThreadF__ should be RTOS__ function PThreadForkHandler named inconsistently, should be PThreadAtForkHandler Again, "PThread" appears in these names to indicate their meaning is not portable. Their existance is, but not their meaning. We have no way of saying "only call this function for pthreads", instead as I understand, we provide a portable interface with drastically varying semantics, such as "do something" vs. "do nothing". Given that Init in ThreadPThread.m3 is private, we could probably take the stackbase parameter from the caller instead. The call to AtFork should probably follow InitWithStackbase, in case it fails under low memory, the Die/assert more likely to "work". I'm not completely happy making RTOS public. So maybe ThreadF? I had gone to RTOS because SetNextForkWillExec required LockHeap/UnlockHeap. But for now they don't exist. Maybe a new interface? ThreadPThread.i3, but is still present on all platforms, so mostly portable can call it. ThreadPThread.AtFork? PThread.AtFork? That seems right. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 17:21:13 +0000 Subject: Re: [M3devel] cvsup fix refinements diffs attached For now I've removed the Get/SetNextForkWillExec. I also think truly get/set is right, not inc/dec. It was confusing enough to interpret and initialize the value. And it is also not clear what the default should be. The usual behavor is exec does follow fork. The safer default if the handlers work ok is to assume exec will not follow. It is a perf hint or a don't change how things were hint? Hint to speed up fork+exec or hint to avoid running the new code that might not work? (For now I've removed the Get/SetNextForkWillExec.) They'd have to be implemented three times, and I had trouble defining them. Obviously Win32/userthreads just return a hardcoded true, but it is hard to explain from the caller of Get's point of view. Maybe: SetNextForkWillExec GetNextForkWillExec and then: PThreadAtForkNeeded() = GetNextForkWillExec = FALSE ? That is, someone is who is calling fork and possibly exec, can immediately tell what these functions mean. But the implementer of an atfork handler has to do quite a semantic translation I think. I wrote it backwards the first time. That either implies it is confusing, or I wasn't thinking. If it really is a problem to run this stuff when exec will follow, we should know quickly, as building cm3 is an aggressive user of this code. This version has worked repeatedly on Darwin. I didn't test user threads but it is structured such as to make that obviously work. The cvsup changes are in pthreaad_atfork handlers, so no change for userthreads. I didn't test on others but confidence is quite high now. description of changes at least where they are hard to read: m3core: Init split into Init and InitWithStackBase, We have to provide the old stack base in the child fork handler instead of estimating from the address of a local. I don't know what stack is used, maybe the caller of fork? i.e. we might have eaten a fair amount of stack and there could be arbitrary references above the current stack. WeakRefFromRef split into WeakRefFromRef and StartWeakCleaner There is a resulting double lock in WeakRefFromRef, every time. Could instead copy/paste more code around, i.e.: EVAL Thread.Fork(NEW(Thread.Closure, apply := WeakCleaner)); - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Fri, 19 Mar 2010 16:01:03 +0000 Subject: [M3devel] cvsup fix refinements refinements: new public functions in m3core: TYPE PThreadForkHandler = PROCEDURE(); <* EXTERNAL RTOS__PThreadAtFork *> PROCEDURE PThreadAtFork(prep, parent, child: PThreadForkHandler): INTEGER; (That's not a change yet.) PROCEDURE SetNextForkWillExec(value: BOOLEAN); PROCEDURE GetNextForkWillExec(): BOOLEAN; These are only an optimization and should perhaps be removed? They should be used under LockHeap/UnlockHeap? which wasn't previously public. This way, existing fork/exec paths not affected, though maybe though might as well ought to be? could go in any of ThreadF, RTOS, RTProcess? Should be called under RTOS.LockHeap? Therefore I put in RTOS. Also helps that RTOS is exported from *Thread.m3. RTOS not previously public. Should Get/Set be Inc/Dec? (therefore just Inc with +1,-1?) I implemented them that way: true incs, false decs. Or don't bother with them at all? Furthermore, in RTCollector, instead of claiming the threads aren't started, if they were started, restart them. This makes more sense for the weak cleaner to me, and seems reasonable for the other two. Diffs not yet available. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 20 04:17:07 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 20 Mar 2010 03:17:07 +0000 Subject: [M3devel] cvsup In-Reply-To: <1268916971.2586.3.camel@faramir.m3w.org> References: ,,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , <1268828983.2906.0.camel@faramir.m3w.org>, , <1268916971.2586.3.camel@faramir.m3w.org> Message-ID: Dead as in was working, but no longer? Please elaborate if not working. What does it do? What platform? Can you please debug it? I been doing mostly RTIO.PutText/Flush anyway. I tried several "debuggers": gdb, m3gdb, dbx, couldn't get much out of any. - Jay > Subject: RE: [M3devel] cvsup > From: dragisha at m3w.org > To: jay.krell at cornell.edu > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > Date: Thu, 18 Mar 2010 13:56:11 +0100 > > Client, it's what I am using. And it's dead as we speak. > > Last worked before I pulled/built RC4 > > > On Thu, 2010-03-18 at 10:09 +0000, Jay K wrote: > > Dragisha, Really? The server? With the -C flag and/or -f? > > > > Evidence is very very very very very good as to what is going on here. > > It is all related to "fork1" vs. "forkall". > > fork1 is the overwhelming usual behavior (Solaris has an option), > > and basically *any* library that uses pthreads is obligated to use > > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > > > The application cannot do things, unless > > the library exposes its internal locks. The Posix spec for > > pthread_atfork explains the problem. > > > > Consider C code that links in Modula-3 code. > > C code is fairly free to use the fork + do work model. It isn't very > > common, > > but it does exist, and people have gone to extra length to keep it > > viable. > > bash and ccache I believe both use this model. > > > > > > Granted, if you had your own kernel thread implementation, it might > > have addressed this. > > > > > > I have a version now that has served over 1000 requests, similar to > > what I sent, but a bit reduced. The main change was I just called > > pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead > > of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just > > Linux. This version also doesn't expose the ForkProcessAndAllThreads > > functions. > > The client has to restart its threads, or in the cvsupd case, don't > > depend on the dispatcher thread to survive fork. > > I want to still try out the "bigger" change, but with the simpler > > lock/unlock. > > And test on Solaris and Darwin. > > > > > > I think for SuspendOthers to not work is not surprising. > > The threads can still hold a few locks even if suspended, I'm pretty > > sure. > > The point is to fork in a "controlled" fashion -- all locks held, and > > in > > the right order, so that both parent and child can release them. > > > > > > Taking "all" the locks in Prepare is more reliable. > > > > > > I do wonder if really "all" locks need to be taken. > > I only take the static ones declared in PThreadThread.c. > > > > - Jay > > > > > > > Subject: Re: [M3devel] cvsup > > > From: dragisha at m3w.org > > > To: jay.krell at cornell.edu > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > > > Yes, I can. I am building it regularly, from version to version, at > > > least few times a year. And it also worked with my ancient kernel > > thread > > > implementation. Was one of stress tests. > > > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > > > (Can anyone claim to have seen cvsup server work with kernel > > threads?) > > > -- > > > Dragi?a Duri? > > > > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 20 09:48:05 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 20 Mar 2010 08:48:05 +0000 Subject: [M3devel] pthread_atfork with user threads? Message-ID: Or should the userthreads version implement itself and require user to call RTProcess.Fork()? You know, we have this odd but ok/useful state: Even though we support userthreads, every system has pthreads. I still have to verify that all systems have pthread_atfork, but I assume so. Also funny thing btw, pthreads, userthreads, win32 threads, the code will be highly shared. They will call RegisterForkHandlers. The fork handlers will be same or similar. Specifically in ThreadPThread.c and ThreadPosix.c -- reinitialize. For ThreadPosix that should achieve "don't switch to any preexisting thread", which is equivalent to "kill all other threads". implied question: Call it AtFork or RegisterForkHandlers or ? RTProcessC.c typedef void (*ForkHandler)(void); #ifndef _WIN32 #include #include #include #endif /* NOTE: Even userthreads now depends * on availability of pthreads. * This can be fixed if need be. */ int RTProcess__RegisterForkHandlers( ForkHandler prepare, ForkHandler parent, ForkHandler child) { #ifdef _WIN32 return 0; #else while (1) { int i = pthread_atfork(prepare, parent, child); if (i != EAGAIN) return i; sleep(0); } #endif } -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 20 18:53:49 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 20 Mar 2010 17:53:49 +0000 Subject: [M3devel] atomics AMD64_DARWIN failures? Message-ID: new source -> compiling AtomicBoolean.m3 ../AMD64_DARWIN/AtomicBoolean.m3 => ../src/atomic/Atomic.mg: In function 'AtomicBoolean__Load': ../AMD64_DARWIN/AtomicBoolean.m3 => ../src/atomic/Atomic.mg:0: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicBoolean.mc new source -> compiling AtomicChar.m3 ../AMD64_DARWIN/AtomicChar.m3 => ../src/atomic/Atomic.mg: In function 'AtomicChar__Load': ../AMD64_DARWIN/AtomicChar.m3 => ../src/atomic/Atomic.mg:0: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicChar.mc new source -> compiling AtomicInteger.m3 ../AMD64_DARWIN/AtomicInteger.m3 => ../src/atomic/Atomic.mg: In function 'AtomicInteger__Store': ../AMD64_DARWIN/AtomicInteger.m3 => ../src/atomic/Atomic.mg:21: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicInteger.mc new source -> compiling AtomicLongint.m3 ../AMD64_DARWIN/AtomicLongint.m3 => ../src/atomic/Atomic.mg: In function 'AtomicLongint__Store': ../AMD64_DARWIN/AtomicLongint.m3 => ../src/atomic/Atomic.mg:21: internal compiler error: Bus error Please submit a full bug report, I'll look into it..first just rebuild from scratch. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 20 19:08:12 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 20 Mar 2010 18:08:12 +0000 Subject: [M3devel] atomics AMD64_DARWIN failures? In-Reply-To: References: Message-ID: Sorry, nevermind, new checkout, new build, works fine. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Date: Sat, 20 Mar 2010 17:53:49 +0000 Subject: [M3devel] atomics AMD64_DARWIN failures? new source -> compiling AtomicBoolean.m3 ../AMD64_DARWIN/AtomicBoolean.m3 => ../src/atomic/Atomic.mg: In function 'AtomicBoolean__Load': ../AMD64_DARWIN/AtomicBoolean.m3 => ../src/atomic/Atomic.mg:0: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicBoolean.mc new source -> compiling AtomicChar.m3 ../AMD64_DARWIN/AtomicChar.m3 => ../src/atomic/Atomic.mg: In function 'AtomicChar__Load': ../AMD64_DARWIN/AtomicChar.m3 => ../src/atomic/Atomic.mg:0: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicChar.mc new source -> compiling AtomicInteger.m3 ../AMD64_DARWIN/AtomicInteger.m3 => ../src/atomic/Atomic.mg: In function 'AtomicInteger__Store': ../AMD64_DARWIN/AtomicInteger.m3 => ../src/atomic/Atomic.mg:21: internal compiler error: Bus error Please submit a full bug report, with preprocessed source if appropriate. See for instructions. m3_backend => 4 m3cc (aka cm3cg) failed compiling: AtomicInteger.mc new source -> compiling AtomicLongint.m3 ../AMD64_DARWIN/AtomicLongint.m3 => ../src/atomic/Atomic.mg: In function 'AtomicLongint__Store': ../AMD64_DARWIN/AtomicLongint.m3 => ../src/atomic/Atomic.mg:21: internal compiler error: Bus error Please submit a full bug report, I'll look into it..first just rebuild from scratch. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 20 21:52:52 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 20 Mar 2010 20:52:52 +0000 Subject: [M3devel] cvsup/fork fix Message-ID: attached is looking pretty good cvsup seems to work on Darwin, kernel threads or user threads I do see: 2010.03.20 13:47:08 PDT [62878]: -0 [0Kin+0Kout] ChannelMux protocol error: Expected EOF, didn't get it but it seems any combination does that. I could try again with no diffs and userthreads. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: atfork4-cvsup.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: atfork4-m3core.txt URL: From jay.krell at cornell.edu Sun Mar 21 02:34:56 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 21 Mar 2010 01:34:56 +0000 Subject: [M3devel] cvsup/fork fix Message-ID: > 2010.03.20 13:47:08 PDT [62878]: -0 [0Kin+0Kout] ChannelMux protocol error: Expected EOF, didn't get it I also see this *occasionally* with user threads and no changes, but very consistently with my changes. I'll probably look into it further. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: cvsup/fork fix Date: Sat, 20 Mar 2010 20:52:52 +0000 attached is looking pretty good cvsup seems to work on Darwin, kernel threads or user threads I do see: 2010.03.20 13:47:08 PDT [62878]: -0 [0Kin+0Kout] ChannelMux protocol error: Expected EOF, didn't get it but it seems any combination does that. I could try again with no diffs and userthreads. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 21 02:40:11 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 21 Mar 2010 01:40:11 +0000 Subject: [M3devel] recent commits Message-ID: commit mail is temporarily down (or is it just me?), so: http://www.modula3.com/cm3/ChangeLog-2010 2010-03-20 20:40 jkrell * m3-libs/m3core/src/runtime/common/RTProcessC.c: new file, empty to start 2010-03-20 07:59 jkrell * m3-libs/m3core/src/Csupport/Common/hand.c: implement set_gt(a,b) as set_lt(b,a) implement set_ge(a,b) as set_le(b,a) Really the frontend or backend should make this transform. 2010-03-20 07:56 jkrell * m3-libs/m3core/src/Csupport/Common/test_hand.c: add some tests for set_lt, le, gt, le; I find this code a bit hard to understand ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 21 08:19:08 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 21 Mar 2010 08:19:08 +0100 Subject: [M3devel] cvsup In-Reply-To: References: ,,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> ,, ,,<1268828983.2906.0.camel@faramir.m3w.org> , ,<1268916971.2586.3.camel@faramir.m3w.org> Message-ID: <1269155948.6829.5.camel@faramir.m3w.org> cvsup client was working for me all the time until latest build - RC4 pulled from local CVS (mirrored by cvsup) on Feb 21 2010. It also worked 5 yrs ago when I used it with my NPTL implementation. What little I understood glimpsing over this discussion, it affects cvsup server? Or is my situation typical? I'll try m3gdb it. Thanks On Sat, 2010-03-20 at 03:17 +0000, Jay K wrote: > Dead as in was working, but no longer? > Please elaborate if not working. What does it do? > What platform? > Can you please debug it? > I been doing mostly RTIO.PutText/Flush anyway. > I tried several "debuggers": gdb, m3gdb, dbx, couldn't get much out of > any. > > - Jay > > > Subject: RE: [M3devel] cvsup > > From: dragisha at m3w.org > > To: jay.krell at cornell.edu > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > Date: Thu, 18 Mar 2010 13:56:11 +0100 > > > > Client, it's what I am using. And it's dead as we speak. > > > > Last worked before I pulled/built RC4 > > > > > > On Thu, 2010-03-18 at 10:09 +0000, Jay K wrote: > > > Dragisha, Really? The server? With the -C flag and/or > -f? > > > > > > Evidence is very very very very very good as to what is going on > here. > > > It is all related to "fork1" vs. "forkall". > > > fork1 is the overwhelming usual behavior (Solaris has an option), > > > and basically *any* library that uses pthreads is obligated to use > > > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > > > > > The application cannot do things, unless > > > the library exposes its internal locks. The Posix spec for > > > pthread_atfork explains the problem. > > > > > > Consider C code that links in Modula-3 code. > > > C code is fairly free to use the fork + do work model. It isn't > very > > > common, > > > but it does exist, and people have gone to extra length to keep it > > > viable. > > > bash and ccache I believe both use this model. > > > > > > > > > Granted, if you had your own kernel thread implementation, it > might > > > have addressed this. > > > > > > > > > I have a version now that has served over 1000 requests, similar > to > > > what I sent, but a bit reduced. The main change was I just called > > > pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, > instead > > > of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just > > > Linux. This version also doesn't expose the > ForkProcessAndAllThreads > > > functions. > > > The client has to restart its threads, or in the cvsupd case, > don't > > > depend on the dispatcher thread to survive fork. > > > I want to still try out the "bigger" change, but with the simpler > > > lock/unlock. > > > And test on Solaris and Darwin. > > > > > > > > > I think for SuspendOthers to not work is not surprising. > > > The threads can still hold a few locks even if suspended, I'm > pretty > > > sure. > > > The point is to fork in a "controlled" fashion -- all locks held, > and > > > in > > > the right order, so that both parent and child can release them. > > > > > > > > > Taking "all" the locks in Prepare is more reliable. > > > > > > > > > I do wonder if really "all" locks need to be taken. > > > I only take the static ones declared in PThreadThread.c. > > > > > > - Jay > > > > > > > > > > Subject: Re: [M3devel] cvsup > > > > From: dragisha at m3w.org > > > > To: jay.krell at cornell.edu > > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > > > > > Yes, I can. I am building it regularly, from version to version, > at > > > > least few times a year. And it also worked with my ancient > kernel > > > thread > > > > implementation. Was one of stress tests. > > > > > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > > > > > (Can anyone claim to have seen cvsup server work with kernel > > > threads?) > > > > -- > > > > Dragi?a Duri? > > > > > > -- > > Dragi?a Duri? > > -- Dragi?a Duri? From jay.krell at cornell.edu Sun Mar 21 09:54:50 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 21 Mar 2010 08:54:50 +0000 Subject: [M3devel] cvsup In-Reply-To: <1269155948.6829.5.camel@faramir.m3w.org> References: ,,,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu>, , , , , , <1268828983.2906.0.camel@faramir.m3w.org>, , , , <1268916971.2586.3.camel@faramir.m3w.org>, , <1269155948.6829.5.camel@faramir.m3w.org> Message-ID: Dragisha, it's still not clear to me if client is still working for not or not, on which platform, and if it doesn't work, what does it do? (Well, NPTL implies Linux.) Yes, mostly we are talking about server here. The server would always hang with kernel threads. But I'm going to look at client more due to this "EOF expected" warning I'm getting. > Or is my situation typical? What is your situation? We have very few reports of any kind, positive or negative. "Dead" could mean you stopped using it. But if it means it stopped working, please, what does it to? If it hangs, put in PutText+Flush calls to find out where it gets to before it hangs? See if it works with user threads (edit m3core/src/thread.quake)? - Jay > Subject: RE: [M3devel] cvsup > From: dragisha at m3w.org > To: jay.krell at cornell.edu > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > Date: Sun, 21 Mar 2010 08:19:08 +0100 > > cvsup client was working for me all the time until latest build - RC4 > pulled from local CVS (mirrored by cvsup) on Feb 21 2010. > > It also worked 5 yrs ago when I used it with my NPTL implementation. > > What little I understood glimpsing over this discussion, it affects > cvsup server? Or is my situation typical? > > I'll try m3gdb it. > > Thanks > > On Sat, 2010-03-20 at 03:17 +0000, Jay K wrote: > > Dead as in was working, but no longer? > > Please elaborate if not working. What does it do? > > What platform? > > Can you please debug it? > > I been doing mostly RTIO.PutText/Flush anyway. > > I tried several "debuggers": gdb, m3gdb, dbx, couldn't get much out of > > any. > > > > - Jay > > > > > Subject: RE: [M3devel] cvsup > > > From: dragisha at m3w.org > > > To: jay.krell at cornell.edu > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > Date: Thu, 18 Mar 2010 13:56:11 +0100 > > > > > > Client, it's what I am using. And it's dead as we speak. > > > > > > Last worked before I pulled/built RC4 > > > > > > > > > On Thu, 2010-03-18 at 10:09 +0000, Jay K wrote: > > > > Dragisha, Really? The server? With the -C flag and/or > > -f? > > > > > > > > Evidence is very very very very very good as to what is going on > > here. > > > > It is all related to "fork1" vs. "forkall". > > > > fork1 is the overwhelming usual behavior (Solaris has an option), > > > > and basically *any* library that uses pthreads is obligated to use > > > > pthread_atfork, be it a C library or the Modula-3 library, etc.. > > > > > > > > The application cannot do things, unless > > > > the library exposes its internal locks. The Posix spec for > > > > pthread_atfork explains the problem. > > > > > > > > Consider C code that links in Modula-3 code. > > > > C code is fairly free to use the fork + do work model. It isn't > > very > > > > common, > > > > but it does exist, and people have gone to extra length to keep it > > > > viable. > > > > bash and ccache I believe both use this model. > > > > > > > > > > > > Granted, if you had your own kernel thread implementation, it > > might > > > > have addressed this. > > > > > > > > > > > > I have a version now that has served over 1000 requests, similar > > to > > > > what I sent, but a bit reduced. The main change was I just called > > > > pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, > > instead > > > > of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just > > > > Linux. This version also doesn't expose the > > ForkProcessAndAllThreads > > > > functions. > > > > The client has to restart its threads, or in the cvsupd case, > > don't > > > > depend on the dispatcher thread to survive fork. > > > > I want to still try out the "bigger" change, but with the simpler > > > > lock/unlock. > > > > And test on Solaris and Darwin. > > > > > > > > > > > > I think for SuspendOthers to not work is not surprising. > > > > The threads can still hold a few locks even if suspended, I'm > > pretty > > > > sure. > > > > The point is to fork in a "controlled" fashion -- all locks held, > > and > > > > in > > > > the right order, so that both parent and child can release them. > > > > > > > > > > > > Taking "all" the locks in Prepare is more reliable. > > > > > > > > > > > > I do wonder if really "all" locks need to be taken. > > > > I only take the static ones declared in PThreadThread.c. > > > > > > > > - Jay > > > > > > > > > > > > > Subject: Re: [M3devel] cvsup > > > > > From: dragisha at m3w.org > > > > > To: jay.krell at cornell.edu > > > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > > > > > > > Yes, I can. I am building it regularly, from version to version, > > at > > > > > least few times a year. And it also worked with my ancient > > kernel > > > > thread > > > > > implementation. Was one of stress tests. > > > > > > > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > > > > > > > (Can anyone claim to have seen cvsup server work with kernel > > > > threads?) > > > > > -- > > > > > Dragi?a Duri? > > > > > > > > -- > > > Dragi?a Duri? > > > > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 21 10:33:11 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 21 Mar 2010 10:33:11 +0100 Subject: [M3devel] cvsup In-Reply-To: References: ,,,, <2F92E7F4-958F-4653-B3F2-384CA03058AB@cs.purdue.edu> ,,, ,,,<1268828983.2906.0.camel@faramir.m3w.org> ,, ,,<1268916971.2586.3.camel@faramir.m3w.org> , ,<1269155948.6829.5.camel@faramir.m3w.org> Message-ID: <1269163991.6829.15.camel@faramir.m3w.org> Few versions ago (May or July, when I last tried) cvsup started to behave ill - it just stops giving any output while memory usage is skyrocketing. With that version, I've synced "incrementally". When it locks, I kill it, then start over. With latest version, one from Feb 21, this does not help anymore. It just stops working (I've not observed if it leaks memory as former version). Half an hour ago I've reinstalled rpm's for one of this former versions I've had on my system. cvsup again sort-of-works. At least it finishes syncing after few restarts. This is what I get with -v: faramir:dragisha/pts/6: ~% cvsup -v CVSup client, non-GUI version Copyright 1996-2003 John D. Polstra Software version: 2009-08-03 16:02:04 Protocol version: 17.0 Operating system: LINUXLIBC6 It looks, for me, like client is culprit. Someone maybe can follow this even further back to find when and where. I don't think kernel threads are so guilty of this behaviour. I will build RC cvs head now. On Sun, 2010-03-21 at 08:54 +0000, Jay K wrote: > Dragisha, it's still not clear to me if client is still working for > not or not, > on which platform, and if it doesn't work, what does it do? > (Well, NPTL implies Linux.) > Yes, mostly we are talking about server here. > The server would always hang with kernel threads. > But I'm going to look at client more due to this "EOF expected" > warning I'm getting. > > > > Or is my situation typical? > > > What is your situation? > We have very few reports of any kind, positive or negative. > > > "Dead" could mean you stopped using it. > But if it means it stopped working, please, what does it to? > If it hangs, put in PutText+Flush calls to find out where > it gets to before it hangs? > See if it works with user threads (edit m3core/src/thread.quake)? > > > - Jay > > > Subject: RE: [M3devel] cvsup > > From: dragisha at m3w.org > > To: jay.krell at cornell.edu > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > Date: Sun, 21 Mar 2010 08:19:08 +0100 > > > > cvsup client was working for me all the time until latest build - > RC4 > > pulled from local CVS (mirrored by cvsup) on Feb 21 2010. > > > > It also worked 5 yrs ago when I used it with my NPTL implementation. > > > > What little I understood glimpsing over this discussion, it affects > > cvsup server? Or is my situation typical? > > > > I'll try m3gdb it. > > > > Thanks > > > > On Sat, 2010-03-20 at 03:17 +0000, Jay K wrote: > > > Dead as in was working, but no longer? > > > Please elaborate if not working. What does it do? > > > What platform? > > > Can you please debug it? > > > I been doing mostly RTIO.PutText/Flush anyway. > > > I tried several "debuggers": gdb, m3gdb, dbx, couldn't get much > out of > > > any. > > > > > > - Jay > > > > > > > Subject: RE: [M3devel] cvsup > > > > From: dragisha at m3w.org > > > > To: jay.krell at cornell.edu > > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > > Date: Thu, 18 Mar 2010 13:56:11 +0100 > > > > > > > > Client, it's what I am using. And it's dead as we speak. > > > > > > > > Last worked before I pulled/built RC4 > > > > > > > > > > > > On Thu, 2010-03-18 at 10:09 +0000, Jay K wrote: > > > > > Dragisha, Really? The server? With the -C flag and/or > > > -f? > > > > > > > > > > Evidence is very very very very very good as to what is going > on > > > here. > > > > > It is all related to "fork1" vs. "forkall". > > > > > fork1 is the overwhelming usual behavior (Solaris has an > option), > > > > > and basically *any* library that uses pthreads is obligated to > use > > > > > pthread_atfork, be it a C library or the Modula-3 library, > etc.. > > > > > > > > > > The application cannot do things, unless > > > > > the library exposes its internal locks. The Posix spec for > > > > > pthread_atfork explains the problem. > > > > > > > > > > Consider C code that links in Modula-3 code. > > > > > C code is fairly free to use the fork + do work model. It > isn't > > > very > > > > > common, > > > > > but it does exist, and people have gone to extra length to > keep it > > > > > viable. > > > > > bash and ccache I believe both use this model. > > > > > > > > > > > > > > > Granted, if you had your own kernel thread implementation, it > > > might > > > > > have addressed this. > > > > > > > > > > > > > > > I have a version now that has served over 1000 requests, > similar > > > to > > > > > what I sent, but a bit reduced. The main change was I just > called > > > > > pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, > > > instead > > > > > of using SuspendOthers. Haven't tested on Solaris/Darwin yet, > just > > > > > Linux. This version also doesn't expose the > > > ForkProcessAndAllThreads > > > > > functions. > > > > > The client has to restart its threads, or in the cvsupd case, > > > don't > > > > > depend on the dispatcher thread to survive fork. > > > > > I want to still try out the "bigger" change, but with the > simpler > > > > > lock/unlock. > > > > > And test on Solaris and Darwin. > > > > > > > > > > > > > > > I think for SuspendOthers to not work is not surprising. > > > > > The threads can still hold a few locks even if suspended, I'm > > > pretty > > > > > sure. > > > > > The point is to fork in a "controlled" fashion -- all locks > held, > > > and > > > > > in > > > > > the right order, so that both parent and child can release > them. > > > > > > > > > > > > > > > Taking "all" the locks in Prepare is more reliable. > > > > > > > > > > > > > > > I do wonder if really "all" locks need to be taken. > > > > > I only take the static ones declared in PThreadThread.c. > > > > > > > > > > - Jay > > > > > > > > > > > > > > > > Subject: Re: [M3devel] cvsup > > > > > > From: dragisha at m3w.org > > > > > > To: jay.krell at cornell.edu > > > > > > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com > > > > > > Date: Wed, 17 Mar 2010 13:29:43 +0100 > > > > > > > > > > > > Yes, I can. I am building it regularly, from version to > version, > > > at > > > > > > least few times a year. And it also worked with my ancient > > > kernel > > > > > thread > > > > > > implementation. Was one of stress tests. > > > > > > > > > > > > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote: > > > > > > > > > > > > > > (Can anyone claim to have seen cvsup server work with > kernel > > > > > threads?) > > > > > > -- > > > > > > Dragi?a Duri? > > > > > > > > > > -- > > > > Dragi?a Duri? > > > > > > -- > > Dragi?a Duri? > > -- Dragi?a Duri? From dragisha at m3w.org Sun Mar 21 18:27:01 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 21 Mar 2010 18:27:01 +0100 Subject: [M3devel] cvsup (fix) state? Message-ID: <1269192421.6829.17.camel@faramir.m3w.org> I've just pulled latest cm3 RC and built up to cvsup... Started client, and it stalls - again - sometimes during process. Is this expected? Fix talk is still talk only? -- Dragi?a Duri? From Highjinks at gmx.com Tue Mar 23 21:02:32 2010 From: Highjinks at gmx.com (Chris) Date: Tue, 23 Mar 2010 21:02:32 +0100 (CET) Subject: [M3devel] Moving ahead with Modula3. Making my code safe. Message-ID: <20100323160650.74e96922.Highjinks@gmx.com> Alright, I'm finally comfortable enough with Modula3 to start doing some serious work with it. However, I'm now in the process of learning how to do things the Modula3 way. Or probably more correctly, writing things that are easily managed by other Modula3 developers. What is the "proper" Modula3 way of taking an unsafe interface to an external library and making it "safe" for the purposes of my fellow Modula3 developers? I know how I would it with Ada and GNAT GPL; but even though the languages are similiar, the environments are quite a bit different. The same thing goes for pretty much everything else. I really want to nail down my libraries and make them as bulletproof as possible. I'd like to make everything as exception free as possible. This means very few uses of the "RAISES" keyword. If a properly defined function returns a predefined error, I dont consider that exceptional. I typically consider any undefined behavior to be exceptional. Also, does Modula3 provide any facilities for Synchronous threads and/or State Machines/Automata, or do these all have to be coded manually? One last question, in regards to threads...how do I get access to Atomic ops? Typically I like to prefetch a dataset into the processors data cache and do most of my operations there. I can doubletime it on a multicore processor, assuming I dont have to use Mutexes and such. Any tips, pointers, or articles you might recommend? -- Chris From wagner at elegosoft.com Wed Mar 24 17:14:54 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Wed, 24 Mar 2010 17:14:54 +0100 Subject: [M3devel] Fwd: m3 Message-ID: <20100324171454.u9gw7n1bz4kcwcs8@mail.elegosoft.com> Any takers? All detail information missing so far :-( Olaf ----- Forwarded message from sunglee726 at gmail.com ----- Date: Wed, 24 Mar 2010 06:01:40 -0500 From: Sung Lee Reply-To: Sung Lee Subject: m3 To: m3-support at elego.de When I compile a modula-3 program and run it, I get an error saying the MSVCR90.dll was not found. What should I do? ----- End forwarded message ----- -------------- next part -------------- When I compile a modula-3 program and run it, I get an error saying the MSVCR90.dll was not found. What should I do? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Mar 27 04:35:17 2010 From: jay.krell at cornell.edu (Jay K) Date: Sat, 27 Mar 2010 03:35:17 +0000 Subject: [M3devel] interface UserThread? Message-ID: Should we offer, say, interface UserThread? One could say IMPORT UserThread AS Thread. It's pretty easy, on Posix. Except there isn't just Thread.T exposed by the thread library. There is e.g. Scheduler, SchedulerPosix. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Sat Mar 27 16:38:40 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Sat, 27 Mar 2010 16:38:40 +0100 Subject: [M3devel] interface UserThread? In-Reply-To: References: Message-ID: <20100327163840.81nebtotwkws0488@mail.elegosoft.com> Quoting Jay K : > Should we offer, say, interface UserThread? > One could say IMPORT UserThread AS Thread. > > It's pretty easy, on Posix. > Except there isn't just Thread.T exposed by the thread library. > There is e.g. Scheduler, SchedulerPosix. You'd like to mix those interfaces? Let's go one step back. What we would actually like to have is a simple switch for system and user threads, for example by just linking against another library. This isn't possible in the current implementation, because _everything_ would have to be recompiled, IIRC. I'm not sure about the reason anymore though. Why can't we have all thread-specific calls pass through one library interface? Does the compiler need to produce different code for some calls? Is it just too inefficient for some functions? If we cannot achieve something simpler that would not need changing the code, then I'd consider a new interface for user threads an option. I don't really like it, because Thread is a standard interface of the language, and we should be very careful to change anything there. At least we should agree that it's worth the efforts. Any opinions? Olaf From wagner at elegosoft.com Sat Mar 27 17:05:53 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Sat, 27 Mar 2010 17:05:53 +0100 Subject: [M3devel] Moving ahead with Modula3. Making my code safe. In-Reply-To: <20100323160650.74e96922.Highjinks@gmx.com> References: <20100323160650.74e96922.Highjinks@gmx.com> Message-ID: <20100327170553.wn2dv2t3448880cs@mail.elegosoft.com> Quoting Chris : > Alright, I'm finally comfortable enough with Modula3 to start doing > some serious work with it. That sounds promising :-) > However, I'm now in the process of learning how to do things the > Modula3 way. Or probably more correctly, writing things that are > easily managed by other Modula3 developers. > > What is the "proper" Modula3 way of taking an unsafe interface to an > external library and making it "safe" for the purposes of my fellow > Modula3 developers? I know how I would it with Ada and GNAT GPL; > but even though the languages are similiar, the environments are > quite a bit different. I don't think you can really make external C and C++ code safe in the sense that Modula-3 code is safe. What you can do is to make sure that the types are mapped correctly, there are no memory leaks and memory overwrites, that all parameters actually are in the expected range, and the returned values are interpreted correctly in the enclosing M3 code. Interfaces to C or C++ are never safe. You can do your best to make the module that uses them safe, i.e. there should be no undetected runtime error or unnoticed side-effect by calling methods of your module. > The same thing goes for pretty much everything else. I really want > to nail down my libraries and make them as bulletproof as possible. Yes, that's the general idea :-) > I'd like to make everything as exception free as possible. This > means very few uses of the "RAISES" keyword. If a properly defined > function returns a predefined error, I dont consider that > exceptional. I typically consider any undefined behavior to be > exceptional. This depends. Exceptions, as the name suggests, should be used for exceptional situations that occur rather infrequently. They may also impose some performance penalty depending on their implementation. The usual, error-free code path should not involve any exceptions. > Also, does Modula3 provide any facilities for Synchronous threads > and/or State Machines/Automata, or do these all have to be coded > manually? Offhand, I don't know of any modules for coroutines or finite state machines. Regarding lexical analysis, you will surely find some FSM-based scanner generators in the compiler tools (caltech-parser). > One last question, in regards to threads...how do I get access to > Atomic ops? Typically I like to prefetch a dataset into the > processors data cache and do most of my operations there. I can > doubletime it on a multicore processor, assuming I dont have to use > Mutexes and such. Atomic operations on processor level are or course very platform specific, and are not exposed in the standard libraries. The standard way to synchronize resource access and make certain operations atomic from an external view is to use mutexes, which are defined in the Thread interface. If this is too high-level or inefficient for certain purposes, Antony Hosking has recently added some generic atomics support in m3core/src/atomic/Atomic.ig. I'm pretty sure it is only in the current head though, and don't know if it's really already implemented on all target platforms. > Any tips, pointers, or articles you might recommend? The language definition as well as a tutorial and a bibliography are online, though I'm afraid that several of the links to external resources are outdated. You surely have browsed through the documentation section of http://www.opencm3.net/? Hope this helps, Olaf From wagner at elegosoft.com Sat Mar 27 17:28:52 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Sat, 27 Mar 2010 17:28:52 +0100 Subject: [M3devel] m3 In-Reply-To: <91819bfb1003240401h6ee7d8fdr70cd68c174cc5d9@mail.gmail.com> References: <91819bfb1003240401h6ee7d8fdr70cd68c174cc5d9@mail.gmail.com> Message-ID: <20100327172852.z6f981if40ww4sok@mail.elegosoft.com> Quoting Sung Lee : > When I compile a modula-3 program and run it, I get an error saying the > MSVCR90.dll was not found. What should I do? Sorry for the delay, I've been very busy again during the week. I forwarded your help request to the m3devel mailing list, but nobody there seems to have been eager to act on it, too. This could be due to the little information you provide though. What exact version of Modula-3 are you using? What commands did you issue, and what C compiler/linker from Microsoft have you installed? It may simply be that the dll in question is not in your path, but I'm not very familiar with Windows systems, so this is just a guess. If you provide more details, others may be able to help. You may also want to create a trac ticket: https://projects.elego.de/cm3/newticket Be sure to fill out all the fields with as much information as possible. Best regards, Olaf From wagner at elegosoft.com Sat Mar 27 18:46:59 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Sat, 27 Mar 2010 18:46:59 +0100 Subject: [M3devel] cvsup/fork fix In-Reply-To: References: Message-ID: <20100327184659.4ipyw1tdicsko0sk@mail.elegosoft.com> Quoting Jay K : > attached is looking pretty good > cvsup seems to work on Darwin, kernel threads or user threads > > I do see: > 2010.03.20 13:47:08 PDT [62878]: -0 [0Kin+0Kout] ChannelMux protocol > error: Expected EOF, didn't get it > > but it seems any combination does that. > I could try again with no diffs and userthreads. What's the status of the fork/threads fixes needed to make CVSup work again? Any chance to get something into the release branch in order to go ahead with RC5? There was also an issue reported with the client. I tried to run it to update the CM3 repository, and it stalls immediately: % /usr/local/cm3/bin/cvsup -g -L 2 -P m -l LOCK_cvsup cvsupfile.cm3 Parsing supfile "cvsupfile.cm3" Connecting to birch.elego.de Connected to birch.elego.de Server software version: release_DCVS_1_0_3_rc8_at_old_elego_de Negotiating file attribute support Exchanging collection information Establishing multiplexed-mode data connection If I run it in the FreeBSD gdb though, it doesn't hang and works as expected (apart from the unnecessary SetAttrs operations). This is with the latest sources from the release branch. It's not deterministic though; sometimes it gets on with its work. I'll try again to catch the deadlock in the debugger. Any other ideas? Olaf From jay.krell at cornell.edu Sun Mar 28 03:26:54 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 28 Mar 2010 01:26:54 +0000 Subject: [M3devel] cvsup/fork Message-ID: The "unexpected EOF" warnings in cvsup worry me, but I'm not sure I have a chance of figuring them out particularly soon. I fear I'll have to read the code a fair amount and look for race conditions, given the intermittent behavior. The changes in m3core which make it work better are correct seemingly independently. I'm inclined to just go with them, at least the pthread part. The EOF warning shows up in the current version even, with user threads, about 1 in 10 times. I'm not seeing m3commit mails. Just me? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 28 08:51:27 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 28 Mar 2010 06:51:27 +0000 Subject: [M3devel] cvsup/SigHandler.m3? Message-ID: This whole file SigHandler.m3 exists so that Modula-3 signal handlers are run in a special thread, free to use any Modula-3 facility. Is it really needed? Is it needed when using pthreads? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From mand at elegosoft.com Sun Mar 28 09:31:41 2010 From: mand at elegosoft.com (mand at elegosoft.com) Date: Sun, 28 Mar 2010 09:31:41 +0200 Subject: [M3devel] test mail Message-ID: <20100328093141.hbtyulj8oo008skg@mail.elego.de> checking for relay to non-elego domains. From jay.krell at cornell.edu Sun Mar 28 10:44:39 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 28 Mar 2010 08:44:39 +0000 Subject: [M3devel] cvsup/fork Message-ID: Mostly nevermind. I'm not seeing the EOF warnings on Linux/x86. Maybe they were Darwin only. I'll look into that "later", but based on Linux/x86 I'll proceed with the fixes. - Jay From: jay.krell at cornell.edu To: m3devel at elegosoft.com Subject: cvsup/fork Date: Sun, 28 Mar 2010 01:26:54 +0000 The "unexpected EOF" warnings in cvsup worry me, but I'm not sure I have a chance of figuring them out particularly soon. I fear I'll have to read the code a fair amount and look for race conditions, given the intermittent behavior. The changes in m3core which make it work better are correct seemingly independently. I'm inclined to just go with them, at least the pthread part. The EOF warning shows up in the current version even, with user threads, about 1 in 10 times. I'm not seeing m3commit mails. Just me? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sun Mar 28 11:58:04 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 28 Mar 2010 09:58:04 +0000 Subject: [M3devel] userthreads vs. pthreads performance? Message-ID: I understand that userthreads don't scale across multiple processors. But testing out cvsup lately, things are noticably much much faster with user threads. At least cvsup. I haven't watched the compiler and such. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 28 12:15:57 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 28 Mar 2010 12:15:57 +0200 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: Message-ID: <1269771357.3671.7.camel@faramir.m3w.org> CVSup w/ user threads is not faster because user threads are faster - it is faster because it's buggy behaviour manifests only with kernel threads. IMO. Algorithm-wise - at least on Linux NPTL is O(1) decision making time, ie switching threads. User threads are lots of linear list walking. Figure how efficient it is. Process management "science" has gone long way since "founding fathers" did user threads thingy. d On Sun, 2010-03-28 at 09:58 +0000, Jay K wrote: > I understand that userthreads don't scale across multiple processors. > But testing out cvsup lately, things are noticably much much faster > with user threads. At least cvsup. I haven't watched the compiler and > such. > > > - Jay > > > > > -- Dragi?a Duri? From mika at async.async.caltech.edu Sun Mar 28 18:24:24 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Sun, 28 Mar 2010 09:24:24 -0700 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269771357.3671.7.camel@faramir.m3w.org> References: <1269771357.3671.7.camel@faramir.m3w.org> Message-ID: <20100328162424.670F91A20B7@async.async.caltech.edu> In my tests, programs that acquire a lot of locks (even uncontended ones, I think) run much much faster with user threads than with kernel threads. Or at least they run much (several times) faster compiled with an ancient PM3 than with a recent CM3, and I'm pretty sure the threading is the main difference. (Given that the effect only occurs for programs that acquire a lot of locks.) Mika =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: >CVSup w/ user threads is not faster because user threads are faster - it >is faster because it's buggy behaviour manifests only with kernel >threads. IMO. > >Algorithm-wise - at least on Linux NPTL is O(1) decision making time, ie >switching threads. User threads are lots of linear list walking. Figure >how efficient it is. Process management "science" has gone long way >since "founding fathers" did user threads thingy. > >d > >On Sun, 2010-03-28 at 09:58 +0000, Jay K wrote: >> I understand that userthreads don't scale across multiple processors. >> But testing out cvsup lately, things are noticably much much faster >> with user threads. At least cvsup. I haven't watched the compiler and >> such. >> >> >> - Jay >> >> >> >> >> >-- >Dragi??a Duri?? From hosking at cs.purdue.edu Sun Mar 28 18:55:36 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 28 Mar 2010 12:55:36 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <20100328162424.670F91A20B7@async.async.caltech.edu> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> Message-ID: <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> Yes, we need to implement thin locks and biased locks. These have minimal overhead for locks held mostly by just one thread. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 28 Mar 2010, at 12:24, Mika Nystrom wrote: > > In my tests, programs that acquire a lot of locks (even uncontended ones, > I think) run much much faster with user threads than with kernel threads. > Or at least they run much (several times) faster compiled with an ancient > PM3 than with a recent CM3, and I'm pretty sure the threading is the > main difference. (Given that the effect only occurs for programs that > acquire a lot of locks.) > > Mika > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: >> CVSup w/ user threads is not faster because user threads are faster - it >> is faster because it's buggy behaviour manifests only with kernel >> threads. IMO. >> >> Algorithm-wise - at least on Linux NPTL is O(1) decision making time, ie >> switching threads. User threads are lots of linear list walking. Figure >> how efficient it is. Process management "science" has gone long way >> since "founding fathers" did user threads thingy. >> >> d >> >> On Sun, 2010-03-28 at 09:58 +0000, Jay K wrote: >>> I understand that userthreads don't scale across multiple processors. >>> But testing out cvsup lately, things are noticably much much faster >>> with user threads. At least cvsup. I haven't watched the compiler and >>> such. >>> >>> >>> - Jay >>> >>> >>> >>> >>> >> -- >> Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Sun Mar 28 20:32:22 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 28 Mar 2010 20:32:22 +0200 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> Message-ID: <1269801143.3671.17.camel@faramir.m3w.org> Fast glance shows there is no overhead in code? Just passthrough of MUTEX field to pthread_mutex_lock()? And inefficiency comes from there? On Sun, 2010-03-28 at 12:55 -0400, Tony Hosking wrote: > Yes, we need to implement thin locks and biased locks. These have > minimal overhead for locks held mostly by just one thread. -- Dragi?a Duri? From mika at async.async.caltech.edu Sun Mar 28 20:57:40 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Sun, 28 Mar 2010 11:57:40 -0700 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269801143.3671.17.camel@faramir.m3w.org> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> <1269801143.3671.17.camel@faramir.m3w.org> Message-ID: <20100328185740.38AFA1A20B9@async.async.caltech.edu> Yep, sounds right. I was profiling some other thread-using code that slowed down enormously because of pthreads and it turned out the program was spending ~95% of its time in accessing the thread locals via one of the pthread_ functions. (The overhead of entering the kernel.) Mika =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: >Fast glance shows there is no overhead in code? Just passthrough of >MUTEX field to pthread_mutex_lock()? And inefficiency comes from there? > >On Sun, 2010-03-28 at 12:55 -0400, Tony Hosking wrote: >> Yes, we need to implement thin locks and biased locks. These have >> minimal overhead for locks held mostly by just one thread. >-- >Dragi??a Duri?? From dragisha at m3w.org Sun Mar 28 21:02:04 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 28 Mar 2010 21:02:04 +0200 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <20100328185740.38AFA1A20B9@async.async.caltech.edu> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> <1269801143.3671.17.camel@faramir.m3w.org> <20100328185740.38AFA1A20B9@async.async.caltech.edu> Message-ID: <1269802924.3671.18.camel@faramir.m3w.org> Which platform? On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > Yep, sounds right. > > I was profiling some other thread-using code that slowed down > enormously > because of pthreads and it turned out the program was spending ~95% > of its time in accessing the thread locals via one of the pthread_ > functions. > (The overhead of entering the kernel.) -- Dragi?a Duri? From hosking at cs.purdue.edu Sun Mar 28 21:08:12 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 28 Mar 2010 15:08:12 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269801143.3671.17.camel@faramir.m3w.org> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> <1269801143.3671.17.camel@faramir.m3w.org> Message-ID: <13B06EBC-1349-4973-BD2B-62C061509471@cs.purdue.edu> Yep. Call. Atomic ops. All expensive. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 28 Mar 2010, at 14:32, Dragi?a Duri? wrote: > Fast glance shows there is no overhead in code? Just passthrough of > MUTEX field to pthread_mutex_lock()? And inefficiency comes from there? > > On Sun, 2010-03-28 at 12:55 -0400, Tony Hosking wrote: >> Yes, we need to implement thin locks and biased locks. These have >> minimal overhead for locks held mostly by just one thread. > -- > Dragi?a Duri? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.async.caltech.edu Sun Mar 28 21:11:16 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Sun, 28 Mar 2010 12:11:16 -0700 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269802924.3671.18.camel@faramir.m3w.org> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> <1269801143.3671.17.camel@faramir.m3w.org> <20100328185740.38AFA1A20B9@async.async.caltech.edu> <1269802924.3671.18.camel@faramir.m3w.org> Message-ID: <20100328191116.CD9C71A20BB@async.async.caltech.edu> Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: >Which platform? > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: >> Yep, sounds right. >> >> I was profiling some other thread-using code that slowed down >> enormously >> because of pthreads and it turned out the program was spending ~95% >> of its time in accessing the thread locals via one of the pthread_ >> functions. >> (The overhead of entering the kernel.) >-- >Dragi??a Duri?? From dragisha at m3w.org Sun Mar 28 21:14:57 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Sun, 28 Mar 2010 21:14:57 +0200 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <20100328191116.CD9C71A20BB@async.async.caltech.edu> References: <1269771357.3671.7.camel@faramir.m3w.org> <20100328162424.670F91A20B7@async.async.caltech.edu> <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> <1269801143.3671.17.camel@faramir.m3w.org> <20100328185740.38AFA1A20B9@async.async.caltech.edu> <1269802924.3671.18.camel@faramir.m3w.org> <20100328191116.CD9C71A20BB@async.async.caltech.edu> Message-ID: <1269803697.3671.19.camel@faramir.m3w.org> I remember reading (long time ago) about how these (FUTEXes) are efficient in LINUX... Can I have your test code to try? On Sun, 2010-03-28 at 12:11 -0700, Mika Nystrom wrote: > Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > >Which platform? > > > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > >> Yep, sounds right. > >> > >> I was profiling some other thread-using code that slowed down > >> enormously > >> because of pthreads and it turned out the program was spending ~95% > >> of its time in accessing the thread locals via one of the pthread_ > >> functions. > >> (The overhead of entering the kernel.) > >-- > >Dragi?a Duri? -- Dragi?a Duri? From mika at async.async.caltech.edu Sun Mar 28 22:33:53 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Sun, 28 Mar 2010 13:33:53 -0700 Subject: [M3devel] Extended Static Checker Message-ID: <20100328203353.8507A1A20BB@async.async.caltech.edu> Hello m3devel, I am curious if the source code for the Modula-3 Extended Static Checker is available anywhere? Or a Linux binary might work for my purposes... I seem to remember they did distribute it at some point... Mika From jay.krell at cornell.edu Sun Mar 28 22:46:01 2010 From: jay.krell at cornell.edu (Jay K) Date: Sun, 28 Mar 2010 20:46:01 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269803697.3671.19.camel@faramir.m3w.org> References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> Message-ID: O(1) scheduling is not a new idea. Just look at NT and probably Solaris and probably all the other non-free systems (AIX, Irix, HP-UX, Tru64, VMS, etc.) Getting thread locals should not require a kernel call. It doesn't on NT. We can optimize this somewhat on most systems with __thread. I had that in briefly. Entering an uncontended pthread mutex should not be expensive -- at least no kernel call, but granted a call and atomic op. Two calls because of the C layer. But user threads pay for a call too of course. Maybe I should profile some of this.. - Jay > From: dragisha at m3w.org > To: mika at async.async.caltech.edu > Date: Sun, 28 Mar 2010 21:14:57 +0200 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > I remember reading (long time ago) about how these (FUTEXes) are > efficient in LINUX... Can I have your test code to try? > > On Sun, 2010-03-28 at 12:11 -0700, Mika Nystrom wrote: > > Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... > > > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > > >Which platform? > > > > > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > > >> Yep, sounds right. > > >> > > >> I was profiling some other thread-using code that slowed down > > >> enormously > > >> because of pthreads and it turned out the program was spending ~95% > > >> of its time in accessing the thread locals via one of the pthread_ > > >> functions. > > >> (The overhead of entering the kernel.) > > >-- > > >Dragi?a Duri? > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Highjinks at gmx.com Sun Mar 28 23:01:31 2010 From: Highjinks at gmx.com (Chris) Date: Sun, 28 Mar 2010 23:01:31 +0200 (CEST) Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: Message-ID: <20100328170619.5d95e6cd.Highjinks@gmx.com> On Sun, 28 Mar 2010 09:58:04 +0000 Jay K wrote: > > I understand that userthreads don't scale across multiple processors. > > But testing out cvsup lately, things are noticably much much faster > > with user threads. At least cvsup. I haven't watched the compiler and such. > > > > > > - Jay In fact user threads, and better yet, Synchronous Threads, are in fact highly scalable when done right. This has been one point of contention for me regarding the current crop of "mature" programming languages. Everything is being shoehorned into the Asynchronous model. The problem is that the Asynchronous model is staring to break down. With more and more processor cores being added, the need for more and more locks and mutexes and semaphores is growing. Not only does this kill performance, it makes problems like deadlock and race conditions much more difficult to prevent. It used to be that user threads, cooperative threads, and Synchronous threads were seen as bottlenecks. That might have been the case back when we were dealing maybe four processors max. But now were looking, in some cases, at machines with four 4 core processors on a MB; and that number is expected to keep growing. Now Cooperative threads and Synchronous threads are outscaling Asynchronous threads, particularly in multimedia applications. The thing that is really needed now is language support and kernel support for the Synchronous model. Here are some reference links, for those who are curious... The Kusp project, which has support for Synchronous threads in the Linux Kernel. http://www.ittc.ku.edu/kurt/ A Wikipedia link on the Esterel programming language. http://en.wikipedia.org/wiki/Esterel Project COSA, which explores the issue a little more in depth. http://www.rebelscience.org/Cosas/COSA.htm The Timber compiler, which is all about Synchronous Programming. For those who care to experiment. http://timber-lang.org/ I realize the OP is about "user threads", but I think it might be good to open up a discussion about "user threads" and the subtopics therein. Modula might benefit from a formal definition of this stuff. IMHO. Anyone else share my interest? -- Chris From jay.krell at cornell.edu Mon Mar 29 03:59:39 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 29 Mar 2010 01:59:39 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <20100328170619.5d95e6cd.Highjinks@gmx.com> References: , <20100328170619.5d95e6cd.Highjinks@gmx.com> Message-ID: People need to know to reduce cross-thread sharing, to reduce lock contention. It is not impossible. There are various conflicting trends though, granted. Windows has long had fibers, but they are pretty impossible to use -- you need your own locking mechanisms, which interact with your own scheduler. There is now "user mode scheduling" in Windows 7 that is easier to use, and the "concurrent runtime" makes it easier to use, along with helping reduce sharing. You should sort of "map/reduce" sorts of things, where each thread can do some of the work independently and there is one last mop-up or sweep pass to collect the data. Consider sorting large data sets for example. You can break up the data into N pieces, sort of them independently on N threads, and then do one last single threaded pass to merge them. "Cloud computing" pushes people toward models of very coarse grained parallelism -- multiple machines, not even the "old coarse grained" multiple processes on the same machine. I think profiling is maybe in order to understand cvsup performance with user threads vs. kernel threads. Usually I ignore performance but the difference is quite noticable in my limited testing. Personally I've only been convinced that kernel threads are generally best and simplest. There are tradeoffs though. Having one mondo scheduler puts a lot of pressure on that one scheduler to figure out things for everyone, whereas within each program there is probably local information that could provide for better scheduling. Basically, taking locks or waiting on objects (files, events, semaphores) is the only way to communicate to a scheduler. It helps a lot, but maybe is not always all that could be done. - Jay > From: Highjinks at gmx.com > To: m3devel at elegosoft.com > Date: Sun, 28 Mar 2010 23:01:31 +0200 > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > On Sun, 28 Mar 2010 09:58:04 +0000 > Jay K wrote: > > > > > I understand that userthreads don't scale across multiple processors. > > > > But testing out cvsup lately, things are noticably much much faster > > > > with user threads. At least cvsup. I haven't watched the compiler and such. > > > > > > > > > > > > - Jay > > In fact user threads, and better yet, Synchronous Threads, are in fact highly scalable when done right. > > This has been one point of contention for me regarding the current crop of "mature" programming languages. Everything is being shoehorned into the Asynchronous model. > > The problem is that the Asynchronous model is staring to break down. With more and more processor cores being added, the need for more and more locks and mutexes and semaphores is growing. Not only does this kill performance, it makes problems like deadlock and race conditions much more difficult to prevent. > > It used to be that user threads, cooperative threads, and Synchronous threads were seen as bottlenecks. That might have been the case back when we were dealing maybe four processors max. But now were looking, in some cases, at machines with four 4 core processors on a MB; and that number is expected to keep growing. Now Cooperative threads and Synchronous threads are outscaling Asynchronous threads, particularly in multimedia applications. > > The thing that is really needed now is language support and kernel support for the Synchronous model. > > Here are some reference links, for those who are curious... > > The Kusp project, which has support for Synchronous threads in the Linux Kernel. > http://www.ittc.ku.edu/kurt/ > > A Wikipedia link on the Esterel programming language. > http://en.wikipedia.org/wiki/Esterel > > Project COSA, which explores the issue a little more in depth. > http://www.rebelscience.org/Cosas/COSA.htm > > The Timber compiler, which is all about Synchronous Programming. For those who care to experiment. > http://timber-lang.org/ > > I realize the OP is about "user threads", but I think it might be good to open up a discussion about "user threads" and the subtopics therein. Modula might benefit from a formal definition of this stuff. IMHO. > > Anyone else share my interest? > > -- > Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Mar 29 04:06:20 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 29 Mar 2010 02:06:20 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <20100328170619.5d95e6cd.Highjinks@gmx.com> References: , <20100328170619.5d95e6cd.Highjinks@gmx.com> Message-ID: http://www.ittc.ku.edu/kurt/ wow, it implies the Linux kernel isn't preemptible...but I did a bit more research..as of 2.6 it is preemtible, configurable, but I don't know what the default is. I failed to mention Erlang, which also pushes process-level concurrency, and much of it. You still need some of locks in these systems, granted. Like file locks. - Jay From: jay.krell at cornell.edu To: highjinks at gmx.com; m3devel at elegosoft.com Subject: RE: [M3devel] userthreads vs. pthreads performance? Date: Mon, 29 Mar 2010 01:59:39 +0000 People need to know to reduce cross-thread sharing, to reduce lock contention. It is not impossible. There are various conflicting trends though, granted. Windows has long had fibers, but they are pretty impossible to use -- you need your own locking mechanisms, which interact with your own scheduler. There is now "user mode scheduling" in Windows 7 that is easier to use, and the "concurrent runtime" makes it easier to use, along with helping reduce sharing. You should sort of "map/reduce" sorts of things, where each thread can do some of the work independently and there is one last mop-up or sweep pass to collect the data. Consider sorting large data sets for example. You can break up the data into N pieces, sort of them independently on N threads, and then do one last single threaded pass to merge them. "Cloud computing" pushes people toward models of very coarse grained parallelism -- multiple machines, not even the "old coarse grained" multiple processes on the same machine. I think profiling is maybe in order to understand cvsup performance with user threads vs. kernel threads. Usually I ignore performance but the difference is quite noticable in my limited testing. Personally I've only been convinced that kernel threads are generally best and simplest. There are tradeoffs though. Having one mondo scheduler puts a lot of pressure on that one scheduler to figure out things for everyone, whereas within each program there is probably local information that could provide for better scheduling. Basically, taking locks or waiting on objects (files, events, semaphores) is the only way to communicate to a scheduler. It helps a lot, but maybe is not always all that could be done. - Jay > From: Highjinks at gmx.com > To: m3devel at elegosoft.com > Date: Sun, 28 Mar 2010 23:01:31 +0200 > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > On Sun, 28 Mar 2010 09:58:04 +0000 > Jay K wrote: > > > > > I understand that userthreads don't scale across multiple processors. > > > > But testing out cvsup lately, things are noticably much much faster > > > > with user threads. At least cvsup. I haven't watched the compiler and such. > > > > > > > > > > > > - Jay > > In fact user threads, and better yet, Synchronous Threads, are in fact highly scalable when done right. > > This has been one point of contention for me regarding the current crop of "mature" programming languages. Everything is being shoehorned into the Asynchronous model. > > The problem is that the Asynchronous model is staring to break down. With more and more processor cores being added, the need for more and more locks and mutexes and semaphores is growing. Not only does this kill performance, it makes problems like deadlock and race conditions much more difficult to prevent. > > It used to be that user threads, cooperative threads, and Synchronous threads were seen as bottlenecks. That might have been the case back when we were dealing maybe four processors max. But now were looking, in some cases, at machines with four 4 core processors on a MB; and that number is expected to keep growing. Now Cooperative threads and Synchronous threads are outscaling Asynchronous threads, particularly in multimedia applications. > > The thing that is really needed now is language support and kernel support for the Synchronous model. > > Here are some reference links, for those who are curious... > > The Kusp project, which has support for Synchronous threads in the Linux Kernel. > http://www.ittc.ku.edu/kurt/ > > A Wikipedia link on the Esterel programming language. > http://en.wikipedia.org/wiki/Esterel > > Project COSA, which explores the issue a little more in depth. > http://www.rebelscience.org/Cosas/COSA.htm > > The Timber compiler, which is all about Synchronous Programming. For those who care to experiment. > http://timber-lang.org/ > > I realize the OP is about "user threads", but I think it might be good to open up a discussion about "user threads" and the subtopics therein. Modula might benefit from a formal definition of this stuff. IMHO. > > Anyone else share my interest? > > -- > Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Mar 29 05:15:41 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 29 Mar 2010 03:15:41 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269803697.3671.19.camel@faramir.m3w.org> References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> Message-ID: > Getting thread locals should not require a kernel call Indeed, on Linux/x86 it does not, looks pretty ok: 00000380 <__pthread_getspecific>: 380: 55 push %ebp 381: 89 e5 mov %esp,%ebp 383: 8b 55 08 mov 0x8(%ebp),%edx 386: 81 fa ff 03 00 00 cmp $0x3ff,%edx 38c: 76 04 jbe 392 <__pthread_getspecific+0x12> 38e: 5d pop %ebp 38f: 31 c0 xor %eax,%eax 391: c3 ret 392: 89 d0 mov %edx,%eax 394: c1 e8 05 shr $0x5,%eax 397: 8d 0c 85 1c 01 00 00 lea 0x11c(,%eax,4),%ecx 39e: 65 8b 01 mov %gs:(%ecx),%eax 3a1: 85 c0 test %eax,%eax 3a3: 74 e9 je 38e <__pthread_getspecific+0xe> 3a5: 8b 04 d5 00 00 00 00 mov 0x0(,%edx,8),%eax 3ac: 85 c0 test %eax,%eax 3ae: 74 de je 38e <__pthread_getspecific+0xe> 3b0: 65 8b 01 mov %gs:(%ecx),%eax 3b3: 83 e2 1f and $0x1f,%edx 3b6: 8b 04 90 mov (%eax,%edx,4),%eax 3b9: 5d pop %ebp 3ba: c3 ret > Entering an uncontended pthread mutex should not be expensive Linux/x86: 00001020 <__pthread_self>: 1020: 55 push %ebp 1021: 89 e5 mov %esp,%ebp 1023: 65 a1 50 00 00 00 mov %gs:0x50,%eax 1029: 5d pop %ebp 102a: c3 ret 102b: 90 nop 102c: 8d 74 26 00 lea 0x0(%esi),%esi pretty lame, five instructions were only two are needed. 000004f0 <__pthread_mutex_lock>: .. too much to read through..but I think no kernel call.. - Jay From: jay.krell at cornell.edu To: dragisha at m3w.org; mika at async.async.caltech.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] userthreads vs. pthreads performance? Date: Sun, 28 Mar 2010 20:46:01 +0000 O(1) scheduling is not a new idea. Just look at NT and probably Solaris and probably all the other non-free systems (AIX, Irix, HP-UX, Tru64, VMS, etc.) Getting thread locals should not require a kernel call. It doesn't on NT. We can optimize this somewhat on most systems with __thread. I had that in briefly. Entering an uncontended pthread mutex should not be expensive -- at least no kernel call, but granted a call and atomic op. Two calls because of the C layer. But user threads pay for a call too of course. Maybe I should profile some of this.. - Jay > From: dragisha at m3w.org > To: mika at async.async.caltech.edu > Date: Sun, 28 Mar 2010 21:14:57 +0200 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > I remember reading (long time ago) about how these (FUTEXes) are > efficient in LINUX... Can I have your test code to try? > > On Sun, 2010-03-28 at 12:11 -0700, Mika Nystrom wrote: > > Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... > > > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > > >Which platform? > > > > > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > > >> Yep, sounds right. > > >> > > >> I was profiling some other thread-using code that slowed down > > >> enormously > > >> because of pthreads and it turned out the program was spending ~95% > > >> of its time in accessing the thread locals via one of the pthread_ > > >> functions. > > >> (The overhead of entering the kernel.) > > >-- > > >Dragi?a Duri? > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Mar 29 10:24:47 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 29 Mar 2010 08:24:47 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269803697.3671.19.camel@faramir.m3w.org> References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> Message-ID: > We can optimize this somewhat on most systems with __thread. I had that in briefly I did a little bit of testing here. __thread takes a few forms, depending on which of -fPIC and -shared you use. Once you do throw in -fPIC and -shared, I have found __thread to be significantly slower on Solaris/sparc and Linux/powerpc32, slower or a wash on Linux/amd64, and twice as fast as pthread_getspecific on Linux/x86. I doesn't appear supported at all on Darwin, though pthread_getspecific are very fast there (albeit not inlined). I didn't test *BSD. My testing was not very scientific. However -fPIC and/or -shared imply a function call to access __thread variables. That's probably the big factor. Without -fPIC/-shared, there is no function call. If you are going to access them multiple times in a function then there is a probably an optimization to be had -- caching their address. If the variables are larger than a pointer, probably then also an optimization to be had. We could do that first thing for certain -- PushFrame could return the address and PopFrame would be much faster. However another angle here is to eliminate PushFrame/PopFrame, by using libunwind. I think we should look into libunwind for the next release. The thread locals will (mostly) remain but accesses to them greatly decline. We compile everything -fPIC and -shared. Some systems (libtool) compile things once that way and once not, providing pairs of libraries, depending on intended use. I should point out also that userthreads have been greatly deoptimized in the current tree (by me, Tony approved), because they used to inline PushFrame/PopFrame, but they don't any longer. (Historically on NT, __declspec(thread) only worked in code in the .exe or statically loaded by the .exe -- that is, not in .dlls loaded with LoadLibrary. However that limitation was removed in Vista. I expect __declspec(thread) is much faster than TlsGetValue, but I assume we'll support pre-Vista for a while longer so not interesting..) - Jay From: jay.krell at cornell.edu To: dragisha at m3w.org; mika at async.async.caltech.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] userthreads vs. pthreads performance? Date: Mon, 29 Mar 2010 03:15:41 +0000 > Getting thread locals should not require a kernel call Indeed, on Linux/x86 it does not, looks pretty ok: 00000380 <__pthread_getspecific>: 380: 55 push %ebp 381: 89 e5 mov %esp,%ebp 383: 8b 55 08 mov 0x8(%ebp),%edx 386: 81 fa ff 03 00 00 cmp $0x3ff,%edx 38c: 76 04 jbe 392 <__pthread_getspecific+0x12> 38e: 5d pop %ebp 38f: 31 c0 xor %eax,%eax 391: c3 ret 392: 89 d0 mov %edx,%eax 394: c1 e8 05 shr $0x5,%eax 397: 8d 0c 85 1c 01 00 00 lea 0x11c(,%eax,4),%ecx 39e: 65 8b 01 mov %gs:(%ecx),%eax 3a1: 85 c0 test %eax,%eax 3a3: 74 e9 je 38e <__pthread_getspecific+0xe> 3a5: 8b 04 d5 00 00 00 00 mov 0x0(,%edx,8),%eax 3ac: 85 c0 test %eax,%eax 3ae: 74 de je 38e <__pthread_getspecific+0xe> 3b0: 65 8b 01 mov %gs:(%ecx),%eax 3b3: 83 e2 1f and $0x1f,%edx 3b6: 8b 04 90 mov (%eax,%edx,4),%eax 3b9: 5d pop %ebp 3ba: c3 ret > Entering an uncontended pthread mutex should not be expensive Linux/x86: 00001020 <__pthread_self>: 1020: 55 push %ebp 1021: 89 e5 mov %esp,%ebp 1023: 65 a1 50 00 00 00 mov %gs:0x50,%eax 1029: 5d pop %ebp 102a: c3 ret 102b: 90 nop 102c: 8d 74 26 00 lea 0x0(%esi),%esi pretty lame, five instructions were only two are needed. 000004f0 <__pthread_mutex_lock>: .. too much to read through..but I think no kernel call.. - Jay From: jay.krell at cornell.edu To: dragisha at m3w.org; mika at async.async.caltech.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] userthreads vs. pthreads performance? Date: Sun, 28 Mar 2010 20:46:01 +0000 O(1) scheduling is not a new idea. Just look at NT and probably Solaris and probably all the other non-free systems (AIX, Irix, HP-UX, Tru64, VMS, etc.) Getting thread locals should not require a kernel call. It doesn't on NT. We can optimize this somewhat on most systems with __thread. I had that in briefly. Entering an uncontended pthread mutex should not be expensive -- at least no kernel call, but granted a call and atomic op. Two calls because of the C layer. But user threads pay for a call too of course. Maybe I should profile some of this.. - Jay > From: dragisha at m3w.org > To: mika at async.async.caltech.edu > Date: Sun, 28 Mar 2010 21:14:57 +0200 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > I remember reading (long time ago) about how these (FUTEXes) are > efficient in LINUX... Can I have your test code to try? > > On Sun, 2010-03-28 at 12:11 -0700, Mika Nystrom wrote: > > Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... > > > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > > >Which platform? > > > > > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > > >> Yep, sounds right. > > >> > > >> I was profiling some other thread-using code that slowed down > > >> enormously > > >> because of pthreads and it turned out the program was spending ~95% > > >> of its time in accessing the thread locals via one of the pthread_ > > >> functions. > > >> (The overhead of entering the kernel.) > > >-- > > >Dragi?a Duri? > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Mon Mar 29 11:38:18 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Mon, 29 Mar 2010 11:38:18 +0200 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: ,<1269771357.3671.7.camel@faramir.m3w.org> ,<20100328162424.670F91A20B7@async.async.caltech.edu> ,<9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> ,<1269801143.3671.17.camel@faramir.m3w.org> ,<20100328185740.38AFA1A20B9@async.async.caltech.edu> ,<1269802924.3671.18.camel@faramir.m3w.org> ,<20100328191116.CD9C71A20BB@async.async.caltech.edu> ,<1269803697.3671.19.camel@faramir.m3w.org> Message-ID: <1269855498.2369.1.camel@faramir.m3w.org> I can be wrong, but what FUTEX operations on Linux can have with pthread_getspecific? Operations are made on FUTEX, we know which and where, no need to ask for pthread_getspecific. Are we still talking about poor lock_mutex performance? On Mon, 2010-03-29 at 08:24 +0000, Jay K wrote: > Once you do throw in -fPIC and -shared, I have found __thread to be > significantly slower on Solaris/sparc and Linux/powerpc32, slower or > a wash on Linux/amd64, and twice as fast as pthread_getspecific on > Linux/x86. > I doesn't appear supported at all on Darwin, though > pthread_getspecific are very fast there (albeit not inlined). > I didn't test *BSD. -- Dragi?a Duri? From jay.krell at cornell.edu Mon Mar 29 15:10:36 2010 From: jay.krell at cornell.edu (Jay K) Date: Mon, 29 Mar 2010 13:10:36 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269855498.2369.1.camel@faramir.m3w.org> References: , , <1269771357.3671.7.camel@faramir.m3w.org>, , <20100328162424.670F91A20B7@async.async.caltech.edu>, , <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, , <1269801143.3671.17.camel@faramir.m3w.org>, , <20100328185740.38AFA1A20B9@async.async.caltech.edu>, , <1269802924.3671.18.camel@faramir.m3w.org>, , <20100328191116.CD9C71A20BB@async.async.caltech.edu>, , <1269803697.3671.19.camel@faramir.m3w.org>, , <1269855498.2369.1.camel@faramir.m3w.org> Message-ID: Original was more general pthread vs. userthread and I thought pthread_getspecific was also indicted. - Jay > From: dragisha at m3w.org > To: jay.krell at cornell.edu > Date: Mon, 29 Mar 2010 11:38:18 +0200 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > I can be wrong, but what FUTEX operations on Linux can have with > pthread_getspecific? > > Operations are made on FUTEX, we know which and where, no need to ask > for pthread_getspecific. > > Are we still talking about poor lock_mutex performance? > > > On Mon, 2010-03-29 at 08:24 +0000, Jay K wrote: > > Once you do throw in -fPIC and -shared, I have found __thread to be > > significantly slower on Solaris/sparc and Linux/powerpc32, slower or > > a wash on Linux/amd64, and twice as fast as pthread_getspecific on > > Linux/x86. > > I doesn't appear supported at all on Darwin, though > > pthread_getspecific are very fast there (albeit not inlined). > > I didn't test *BSD. > -- > Dragi?a Duri? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 29 16:42:17 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 29 Mar 2010 10:42:17 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> Message-ID: Guys, You are looking in the wrong place. The whole good thing about having a language-defined thread model is that we get to implement the thread primitives in the language run-time, and we can take advantage of language-specific semantics. There is no requirement that a Modula-3 mutex even map to a pthread_mutex_t. Java monitors are implemented in modern JVMs so that lock/unlock in the common case doesn't require any calls or any atomic operations! We can do much the same for Modula-3. My group at Purdue has done a fair amount of work on this in the context of Java, and when I can find the time I was hoping to do the same for Modula-3. The only requirement for M3 was that we have proper support for atomic ops in the language upon which to build the synchronisation primitives. We are close to having this now. Let me sketch a design: The common case is that a mutex is only ever manipulated by one thread (i.e., never shared), in which case it suffices to "bias" the mutex to that thread. Locking/unlocking is simply a matter of checking to see that the bias belongs to the current thread. No need for an atomic operation. If the thread already has the bias then locking/unlocking is simply a matter of setting a bit in the mutex. If another thread comes along and needs to lock the mutex then it must first revoke the bias of the owning thread. This can be expensive (assuming it occurs infrequently) and in our case probably means stopping the thread having the bias, revoking the bias, then restarting it. Another case is when a mutex is locked/unlocked by multiple threads but there is never contention (i.e., no thread tries to acquire while another thread holds). In this case we never need a wait queue for the mutex so we can simply store the lock owner in the mutex and test using atomic ops. Spinning is often useful here to avoid needing to inflate if contention ever does arise: if the thread holding the lock gets out of the mutex quickly then the spinner can move in quickly. After some number of spins we generally need to inflate the lock to allocate a wait queue (to avoiding hogging the processor). Finally, the case where many threads are contending on the inflated lock (with wait queue). The only question now is when to deflate. Our current heuristic is to deflate when the last thread releases the lock and notices that there are no other threads waiting. This seems to work well in practice, but of course there are pathological cases. Note that in no case have I mentioned the need for a pthread_mutex (though pthread locks/conditions are used to manage threads that must block on a Java quit queue). We ought to be able to do much the same in Modula-3. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 29 Mar 2010, at 04:24, Jay K wrote: > > We can optimize this somewhat on most systems with __thread. I had that in briefly > > > I did a little bit of testing here. > > > __thread takes a few forms, depending on which of -fPIC and -shared you use. > > Once you do throw in -fPIC and -shared, I have found __thread to be significantly slower on Solaris/sparc and Linux/powerpc32, slower or a wash on Linux/amd64, and twice as fast as pthread_getspecific on Linux/x86. > I doesn't appear supported at all on Darwin, though pthread_getspecific are very fast there (albeit not inlined). > I didn't test *BSD. > My testing was not very scientific. > However -fPIC and/or -shared imply a function call to access __thread variables. > That's probably the big factor. Without -fPIC/-shared, there is no function call. > > > If you are going to access them multiple times in a function then there is a probably an optimization to be had -- caching their address. If the variables are larger than a pointer, probably then also an optimization to be had. > > > We could do that first thing for certain -- PushFrame could return the address and PopFrame would be much faster. > However another angle here is to eliminate PushFrame/PopFrame, by using libunwind. I think we should look into libunwind for the next release. The thread locals will (mostly) remain but accesses to them greatly decline. > > > We compile everything -fPIC and -shared. > Some systems (libtool) compile things once that way and once not, providing pairs of libraries, depending on intended use. > > > I should point out also that userthreads have been greatly deoptimized in the current tree (by me, Tony approved), because they used to inline PushFrame/PopFrame, but they don't any longer. > > > (Historically on NT, __declspec(thread) only worked in code in the .exe or statically loaded by the .exe -- that is, not in .dlls loaded with LoadLibrary. However that limitation was removed in Vista. I expect __declspec(thread) is much faster than TlsGetValue, but I assume we'll support pre-Vista for a while longer so not interesting..) > > > - Jay > > > From: jay.krell at cornell.edu > To: dragisha at m3w.org; mika at async.async.caltech.edu > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] userthreads vs. pthreads performance? > Date: Mon, 29 Mar 2010 03:15:41 +0000 > > > Getting thread locals should not require a kernel call > > Indeed, on Linux/x86 it does not, looks pretty ok: > > 00000380 <__pthread_getspecific>: > 380: 55 push %ebp > 381: 89 e5 mov %esp,%ebp > 383: 8b 55 08 mov 0x8(%ebp),%edx > 386: 81 fa ff 03 00 00 cmp $0x3ff,%edx > 38c: 76 04 jbe 392 <__pthread_getspecific+0x12> > 38e: 5d pop %ebp > 38f: 31 c0 xor %eax,%eax > 391: c3 ret > > 392: 89 d0 mov %edx,%eax > 394: c1 e8 05 shr $0x5,%eax > 397: 8d 0c 85 1c 01 00 00 lea 0x11c(,%eax,4),%ecx > 39e: 65 8b 01 mov %gs:(%ecx),%eax > 3a1: 85 c0 test %eax,%eax > 3a3: 74 e9 je 38e <__pthread_getspecific+0xe> > 3a5: 8b 04 d5 00 00 00 00 mov 0x0(,%edx,8),%eax > 3ac: 85 c0 test %eax,%eax > 3ae: 74 de je 38e <__pthread_getspecific+0xe> > 3b0: 65 8b 01 mov %gs:(%ecx),%eax > 3b3: 83 e2 1f and $0x1f,%edx > 3b6: 8b 04 90 mov (%eax,%edx,4),%eax > 3b9: 5d pop %ebp > 3ba: c3 ret > > > > Entering an uncontended pthread mutex should not be expensive > > Linux/x86: > > 00001020 <__pthread_self>: > 1020: 55 push %ebp > 1021: 89 e5 mov %esp,%ebp > 1023: 65 a1 50 00 00 00 mov %gs:0x50,%eax > 1029: 5d pop %ebp > 102a: c3 ret > 102b: 90 nop > 102c: 8d 74 26 00 lea 0x0(%esi),%esi > > > pretty lame, five instructions were only two are needed. > > > 000004f0 <__pthread_mutex_lock>: > > .. too much to read through..but I think no kernel call.. > > - Jay > > From: jay.krell at cornell.edu > To: dragisha at m3w.org; mika at async.async.caltech.edu > CC: m3devel at elegosoft.com > Subject: RE: [M3devel] userthreads vs. pthreads performance? > Date: Sun, 28 Mar 2010 20:46:01 +0000 > > O(1) scheduling is not a new idea. Just look at NT and probably Solaris and probably all the other non-free systems (AIX, Irix, HP-UX, Tru64, VMS, etc.) > > Getting thread locals should not require a kernel call. It doesn't on NT. We can optimize this somewhat on most systems with __thread. I had that in briefly. > > Entering an uncontended pthread mutex should not be expensive -- at least no kernel call, but granted a call and atomic op. Two calls because of the C layer. > But user threads pay for a call too of course. > > Maybe I should profile some of this.. > > - Jay > > > From: dragisha at m3w.org > > To: mika at async.async.caltech.edu > > Date: Sun, 28 Mar 2010 21:14:57 +0200 > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] userthreads vs. pthreads performance? > > > > I remember reading (long time ago) about how these (FUTEXes) are > > efficient in LINUX... Can I have your test code to try? > > > > On Sun, 2010-03-28 at 12:11 -0700, Mika Nystrom wrote: > > > Well I have run programs on PPC_DARWIN and FreeBSD and seen these sorts of things... > > > > > > =?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?= writes: > > > >Which platform? > > > > > > > >On Sun, 2010-03-28 at 11:57 -0700, Mika Nystrom wrote: > > > >> Yep, sounds right. > > > >> > > > >> I was profiling some other thread-using code that slowed down > > > >> enormously > > > >> because of pthreads and it turned out the program was spending ~95% > > > >> of its time in accessing the thread locals via one of the pthread_ > > > >> functions. > > > >> (The overhead of entering the kernel.) > > > >-- > > > >Dragi?a Duri? > > -- > > Dragi?a Duri? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Mon Mar 29 16:43:21 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 29 Mar 2010 10:43:21 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <1269855498.2369.1.camel@faramir.m3w.org> References: , <1269771357.3671.7.camel@faramir.m3w.org> , <20100328162424.670F91A20B7@async.async.caltech.edu> , <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu> , <1269801143.3671.17.camel@faramir.m3w.org> , <20100328185740.38AFA1A20B9@async.async.caltech.edu> , <1269802924.3671.18.camel@faramir.m3w.org> , <20100328191116.CD9C71A20BB@async.async.caltech.edu> , <1269803697.3671.19.camel@faramir.m3w.org> <1269855498.2369.1.camel@faramir.m3w.org> Message-ID: PS. Our Java work shows that futexes don't really win against the bias/thin/fat lock scheme I sketched previously. On 29 Mar 2010, at 05:38, Dragi?a Duri? wrote: > I can be wrong, but what FUTEX operations on Linux can have with > pthread_getspecific? > > Operations are made on FUTEX, we know which and where, no need to ask > for pthread_getspecific. > > Are we still talking about poor lock_mutex performance? > > > On Mon, 2010-03-29 at 08:24 +0000, Jay K wrote: >> Once you do throw in -fPIC and -shared, I have found __thread to be >> significantly slower on Solaris/sparc and Linux/powerpc32, slower or >> a wash on Linux/amd64, and twice as fast as pthread_getspecific on >> Linux/x86. >> I doesn't appear supported at all on Darwin, though >> pthread_getspecific are very fast there (albeit not inlined). >> I didn't test *BSD. > -- > Dragi?a Duri? From rodney_bates at lcwb.coop Mon Mar 29 17:01:04 2010 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 29 Mar 2010 10:01:04 -0500 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> Message-ID: <4BB0C0B0.3010108@lcwb.coop> Tony Hosking wrote: > Guys, > > You are looking in the wrong place. The whole good thing about having a > language-defined thread model is that we get to implement the thread > primitives in the language run-time, and we can take advantage of > language-specific semantics. There is no requirement that a Modula-3 > mutex even map to a pthread_mutex_t. Java monitors are implemented in > modern JVMs so that lock/unlock in the common case doesn't require any > calls or any atomic operations! We can do much the same for Modula-3. > My group at Purdue has done a fair amount of work on this in the > context of Java, and when I can find the time I was hoping to do the > same for Modula-3. The only requirement for M3 was that we have proper > support for atomic ops in the language upon which to build the > synchronisation primitives. We are close to having this now. Let me > sketch a design: > > The common case is that a mutex is only ever manipulated by one thread Do you mean _almost_ only ever? -----^ Otherwise, why would it have been coded with a mutex at all? > (i.e., never shared), in which case it suffices to "bias" the mutex to > that thread. Locking/unlocking is simply a matter of checking to see > that the bias belongs to the current thread. No need for an atomic > operation. If the thread already has the bias then locking/unlocking is > simply a matter of setting a bit in the mutex. If another thread comes > along and needs to lock the mutex then it must first revoke the bias of > the owning thread. This can be expensive (assuming it occurs > infrequently) and in our case probably means stopping the thread having > the bias, revoking the bias, then restarting it. > > Another case is when a mutex is locked/unlocked by multiple threads but > there is never contention (i.e., no thread tries to acquire while -----------^ and _almost_ never? > another thread holds). In this case we never need a wait queue for the > mutex so we can simply store the lock owner in the mutex and test using > atomic ops. Spinning is often useful here to avoid needing to inflate > if contention ever does arise: if the thread holding the lock gets out > of the mutex quickly then the spinner can move in quickly. After some > number of spins we generally need to inflate the lock to allocate a wait > queue (to avoiding hogging the processor). > > Finally, the case where many threads are contending on the inflated lock > (with wait queue). The only question now is when to deflate. Our > current heuristic is to deflate when the last thread releases the lock > and notices that there are no other threads waiting. This seems to work > well in practice, but of course there are pathological cases. So does the implementation dynamically detect which case holds and choose between the schemes? What are the criteria? > > Note that in no case have I mentioned the need for a pthread_mutex > (though pthread locks/conditions are used to manage threads that must > block on a Java quit queue). > > We ought to be able to do much the same in Modula-3. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > From mika at async.async.caltech.edu Mon Mar 29 17:20:25 2010 From: mika at async.async.caltech.edu (Mika Nystrom) Date: Mon, 29 Mar 2010 08:20:25 -0700 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <4BB0C0B0.3010108@lcwb.coop> References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> <4BB0C0B0.3010108@lcwb.coop> Message-ID: <20100329152025.630591A20C2@async.async.caltech.edu> "Rodney M. Bates" writes: > ... >> >> The common case is that a mutex is only ever manipulated by one thread >Do you mean _almost_ only ever? -----^ Otherwise, why would it have been >coded with a mutex at all? ... In a single-threaded program.... lots of mutexes hiding in various software libraries (e.g., Rd/Wr)... Mika From hosking at cs.purdue.edu Mon Mar 29 19:26:28 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 29 Mar 2010 13:26:28 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <4BB0C0B0.3010108@lcwb.coop> References: , <1269771357.3671.7.camel@faramir.m3w.org>, <20100328162424.670F91A20B7@async.async.caltech.edu>, <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, <1269801143.3671.17.camel@faramir.m3w.org>, <20100328185740.38AFA1A20B9@async.async.caltech.edu>, <1269802924.3671.18.camel@faramir.m3w.org>, <20100328191116.CD9C71A20BB@async.async.caltech.edu>, <1269803697.3671.19.camel@faramir.m3w.org> <4BB0C0B0.3010108@lcwb.coop> Message-ID: <3716771F-D63C-4ABC-8977-3B7F509BEB88@cs.purdue.edu> On 29 Mar 2010, at 11:01, Rodney M. Bates wrote: > > > Tony Hosking wrote: >> Guys, >> You are looking in the wrong place. The whole good thing about having a language-defined thread model is that we get to implement the thread primitives in the language run-time, and we can take advantage of language-specific semantics. There is no requirement that a Modula-3 mutex even map to a pthread_mutex_t. Java monitors are implemented in modern JVMs so that lock/unlock in the common case doesn't require any calls or any atomic operations! We can do much the same for Modula-3. My group at Purdue has done a fair amount of work on this in the context of Java, and when I can find the time I was hoping to do the same for Modula-3. The only requirement for M3 was that we have proper support for atomic ops in the language upon which to build the synchronisation primitives. We are close to having this now. Let me sketch a design: >> The common case is that a mutex is only ever manipulated by one thread > Do you mean _almost_ only ever? -----^ Otherwise, why would it have been > coded with a mutex at all? You might have a thread-safe library that is used by a single-thread application. >> (i.e., never shared), in which case it suffices to "bias" the mutex to that thread. Locking/unlocking is simply a matter of checking to see that the bias belongs to the current thread. No need for an atomic operation. If the thread already has the bias then locking/unlocking is simply a matter of setting a bit in the mutex. If another thread comes along and needs to lock the mutex then it must first revoke the bias of the owning thread. This can be expensive (assuming it occurs infrequently) and in our case probably means stopping the thread having the bias, revoking the bias, then restarting it. >> Another case is when a mutex is locked/unlocked by multiple threads but there is never contention (i.e., no thread tries to acquire while > -----------^ and _almost_ never? > >> another thread holds). In this case we never need a wait queue for the mutex so we can simply store the lock owner in the mutex and test using atomic ops. Spinning is often useful here to avoid needing to inflate if contention ever does arise: if the thread holding the lock gets out of the mutex quickly then the spinner can move in quickly. After some number of spins we generally need to inflate the lock to allocate a wait queue (to avoiding hogging the processor). >> Finally, the case where many threads are contending on the inflated lock (with wait queue). The only question now is when to deflate. Our current heuristic is to deflate when the last thread releases the lock and notices that there are no other threads waiting. This seems to work well in practice, but of course there are pathological cases. > > So does the implementation dynamically detect which case holds and > choose between the schemes? What are the criteria? Short answer, yes. Criteria much as I described. The only trick thing is choosing when to deflate. For more on this topic take a look also at: http://blogs.azulsystems.com/cliff/. Azul independently developed similar strategies. > >> Note that in no case have I mentioned the need for a pthread_mutex (though pthread locks/conditions are used to manage threads that must block on a Java quit queue). >> We ought to be able to do much the same in Modula-3. >> Antony Hosking | Associate Professor | Computer Science | Purdue University >> 305 N. University Street | West Lafayette | IN 47907 | USA >> Office +1 765 494 6001 | Mobile +1 765 427 5484 >> From dragisha at m3w.org Tue Mar 30 01:47:36 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Tue, 30 Mar 2010 01:47:36 +0200 Subject: [M3devel] user vs. kernel Message-ID: <1269906456.3029.4.camel@faramir.m3w.org> I am following this thread, and I have just few points to highlight: 1. battle user vs. kernel threads was finished long time ago. let's not pull back Modula-3 back where it was before last few years of development by doing things backward; 2. bias et al is nice thing, and I believe we will have it in no time; as compared to total development time of cm3. -- Dragi?a Duri? From jay.krell at cornell.edu Tue Mar 30 08:37:40 2010 From: jay.krell at cornell.edu (Jay K) Date: Tue, 30 Mar 2010 06:37:40 +0000 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: <3716771F-D63C-4ABC-8977-3B7F509BEB88@cs.purdue.edu> References: , , <1269771357.3671.7.camel@faramir.m3w.org>, , <20100328162424.670F91A20B7@async.async.caltech.edu>, , <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, , <1269801143.3671.17.camel@faramir.m3w.org>, , <20100328185740.38AFA1A20B9@async.async.caltech.edu>, , <1269802924.3671.18.camel@faramir.m3w.org>, , <20100328191116.CD9C71A20BB@async.async.caltech.edu>, , <1269803697.3671.19.camel@faramir.m3w.org> , , <4BB0C0B0.3010108@lcwb.coop>, <3716771F-D63C-4ABC-8977-3B7F509BEB88@cs.purdue.edu> Message-ID: > The only requirement for M3 was that we have proper support for > atomic ops in the language upon which to build the synchronisation > primitives. We are close to having this now. Or surely implement them in C.. Anyway, I believe m3back now (a few weeks) has parity with or exceeds the gcc backend. Depending on interpretation. The gcc backend only accepts some memory orders. m3back accepts them all and interprets them all as sequential. I have more testing and optimization to do, but I think it is all there and probably works. Optimization in particular is that add/sub/xor don't need to use compare/exchange/retry loops. And load/store could certainly be more efficient such as just using one exchange to do a store, instead of a fence on either side. I think it is also probably reasonable to defined variants that don't return the new value. That way or and and can be provided more efficiently -- without a compare/exchange/retry loop. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Mar 30 14:51:34 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 30 Mar 2010 08:51:34 -0400 Subject: [M3devel] userthreads vs. pthreads performance? In-Reply-To: References: , , <1269771357.3671.7.camel@faramir.m3w.org>, , <20100328162424.670F91A20B7@async.async.caltech.edu>, , <9C08E8EF-721C-4DC0-8150-B411251D0D48@cs.purdue.edu>, , <1269801143.3671.17.camel@faramir.m3w.org>, , <20100328185740.38AFA1A20B9@async.async.caltech.edu>, , <1269802924.3671.18.camel@faramir.m3w.org>, , <20100328191116.CD9C71A20BB@async.async.caltech.edu>, , <1269803697.3671.19.camel@faramir.m3w.org> , , <4BB0C0B0.3010108@lcwb.coop>, <3716771F-D63C-4ABC-8977-3B7F509BEB88@cs.purdue.edu> Message-ID: On 30 Mar 2010, at 02:37, Jay K wrote: > > The only requirement for M3 was that we have proper support for > > atomic ops in the language upon which to build the synchronisation > > primitives. We are close to having this now. > > > Or surely implement them in C.. We want them inline. > Anyway, I believe m3back now (a few weeks) has parity with or exceeds the gcc backend. > Depending on interpretation. > > > The gcc backend only accepts some memory orders. > > > m3back accepts them all and interprets them all as sequential. same as gcc. > > > I have more testing and optimization to do, but I think it is all there > and probably works. Optimization in particular is that add/sub/xor > don't need to use compare/exchange/retry loops. > > > And load/store could certainly be more efficient such as just > using one exchange to do a store, instead of a fence on either side. > > > I think it is also probably reasonable to defined variants that > don't return the new value. That way or and and can be provided > more efficiently -- without a compare/exchange/retry loop. > > > - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 31 02:53:27 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 00:53:27 +0000 Subject: [M3devel] m3core/src/types/Unicode.m3 var to const Message-ID: This is right, right? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 1.txt URL: From hosking at cs.purdue.edu Wed Mar 31 04:39:25 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 30 Mar 2010 22:39:25 -0400 Subject: [M3devel] m3core/src/types/Unicode.m3 var to const In-Reply-To: References: Message-ID: <91EA5B0C-96CA-4C36-B9C1-DEF2BC5D29CB@cs.purdue.edu> Yes, I think that looks OK. The original is very weird M3 code! Also, can you now make it a safe module instead of UNSAFE? (No use of ADR anymore). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 30 Mar 2010, at 20:53, Jay K wrote: > This is right, right? > > - Jay > > > > > > <1.txt> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 31 05:19:57 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 03:19:57 +0000 Subject: [M3devel] m3core/src/types/Unicode.m3 var to const In-Reply-To: <91EA5B0C-96CA-4C36-B9C1-DEF2BC5D29CB@cs.purdue.edu> References: , <91EA5B0C-96CA-4C36-B9C1-DEF2BC5D29CB@cs.purdue.edu> Message-ID: "I think I can". Just use arrays and indices more and pointers to elements less/not at all. Granted, moving the division is a big speed deoptimization since it is no longer by a constant (unless compiler is quite smart and does "partial inlining", and NT386 compiler is not, very few compilers are smart enough to see this). It might be worth a) restoring that small part b) having two functions, one for 2-tuples, one for 3-tuples, or even c) surely we con't need to handcode binary search here in the first place there is some generic reusable version? But that is tangential to var vs. const. I was also considering adding: Int8, UInt8, Int16, UInt16, UInt32, UInt64. In particular, in m3back, I think I'll be using UInt16 in tables -- CodeView 4.0 format uses UINT16 to represent types (CodeView 5.0 uses UINT32). In reality due to padding/alignment, UINT8/16 probably have no space savings, but they will get range checks..but I'll also have to add a range check when I "allocate" a new type. "That" is "why" I was looking in this directory. - Jay From: hosking at cs.purdue.edu Date: Tue, 30 Mar 2010 22:39:25 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] m3core/src/types/Unicode.m3 var to const Yes, I think that looks OK. The original is very weird M3 code! Also, can you now make it a safe module instead of UNSAFE? (No use of ADR anymore). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 30 Mar 2010, at 20:53, Jay K wrote: This is right, right? - Jay <1.txt> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 31 06:41:14 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 04:41:14 +0000 Subject: [M3devel] m3back directions? Message-ID: I'm curious what, if anything, people are interested in in m3back? There are several mostly independent directions: - remove it; use the gcc backend or other (burg, llvm, generate C) - expand to support other targets, AMD64_*, including AMD64_NT m3objfile would need macho/elf support for non-NT - expand to generate good debugging information for Microsoft debuggers - various smaller/larger optimizations inlining? Inlining seems like the most lacking optimization and thinking about it, it doesn't actually seem that hard to do, at least a little bit. I'm thinking for now of working on the debugging information. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 31 10:02:48 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 08:02:48 +0000 Subject: [M3devel] generic binary search in standard Modula-3 library? In-Reply-To: References: , , <91EA5B0C-96CA-4C36-B9C1-DEF2BC5D29CB@cs.purdue.edu>, Message-ID: > c) surely we con't need to handcode binary search here in the first place there is some generic reusable version? The closest precedent I can find is: http://modula3.elegosoft.com/cm3/doc/help/gen_html/libm3/src/sort/ArraySort.ig.html so I propose *something like*: m3core/src/binarysearch/BinarySearch.ig. PROCEDURE BinarySearch(READONLY key: Elem.T; READONLY a: ARRAY OF Elem.T; cmp := Elem.Compare): INTEGER; along with: PROCEDURE LowerBound(READONLY key: Elem.T; READONLY a: ARRAY OF Elem.T; cmp := Elem.Compare): INTEGER; PROCEDURE UpperBound(READONLY key: Elem.T; READONLY a: ARRAY OF Elem.T; cmp := Elem.Compare): INTEGER; PROCEDURE EqualRange(READONLY key: Elem.T; READONLY a: ARRAY OF Elem.T; VAR lower, upper: INTEGER; cmp := Elem.Compare); using -1 for not found? LowerBound, UpperBound, EqualRange are closely related to BinarySearch. When the array contains adjacent equal/equivalent elements, they let you find the first, last or first and last, whereas "regular" BinarySearch finds an arbitrary one. See the C++ STL. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Wed, 31 Mar 2010 03:19:57 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] m3core/src/types/Unicode.m3 var to const "I think I can". Just use arrays and indices more and pointers to elements less/not at all. Granted, moving the division is a big speed deoptimization since it is no longer by a constant (unless compiler is quite smart and does "partial inlining", and NT386 compiler is not, very few compilers are smart enough to see this). It might be worth a) restoring that small part b) having two functions, one for 2-tuples, one for 3-tuples, or even c) surely we con't need to handcode binary search here in the first place there is some generic reusable version? But that is tangential to var vs. const. I was also considering adding: Int8, UInt8, Int16, UInt16, UInt32, UInt64. In particular, in m3back, I think I'll be using UInt16 in tables -- CodeView 4.0 format uses UINT16 to represent types (CodeView 5.0 uses UINT32). In reality due to padding/alignment, UINT8/16 probably have no space savings, but they will get range checks..but I'll also have to add a range check when I "allocate" a new type. "That" is "why" I was looking in this directory. - Jay From: hosking at cs.purdue.edu Date: Tue, 30 Mar 2010 22:39:25 -0400 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] m3core/src/types/Unicode.m3 var to const Yes, I think that looks OK. The original is very weird M3 code! Also, can you now make it a safe module instead of UNSAFE? (No use of ADR anymore). Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On 30 Mar 2010, at 20:53, Jay K wrote: This is right, right? - Jay <1.txt> -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Wed Mar 31 11:37:31 2010 From: dragisha at m3w.org (=?UTF-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Wed, 31 Mar 2010 11:37:31 +0200 Subject: [M3devel] m3back directions? In-Reply-To: References: Message-ID: <1270028251.3114.4.camel@faramir.m3w.org> It is started job and useable right now. IMO - it's prolonged and thriving existence shows directions and maintains secure interface for multiple backends - LLVM for example. Thus said, my opinion is: a) keep it, and make it excellent working x86 backend alternative. b) AMD64, as natural successor to x86 - maybe, if it's not too much work. Even without it - m3back can probably work for many x86 appliances present - for years to come. c) debug - of course it's good idea. But, if it's alternative, one can debug her code using gcc, then switch to m3back build for productivity. d) nothing too big of optimizations, although inlining would be sweet. dd On Wed, 2010-03-31 at 04:41 +0000, Jay K wrote: > I'm curious what, if anything, people are interested in in m3back? > There are several mostly independent directions: > - remove it; use the gcc backend or other (burg, llvm, generate C) > - expand to support other targets, AMD64_*, including AMD64_NT > m3objfile would need macho/elf support for non-NT > - expand to generate good debugging information for Microsoft > debuggers > - various smaller/larger optimizations > inlining? Inlining seems like the most lacking optimization and > thinking about it, it doesn't actually seem that hard to do, at least > a little bit. -- Dragi?a Duri? From wagner at elegosoft.com Wed Mar 31 12:18:30 2010 From: wagner at elegosoft.com (wagner at elegosoft.com) Date: Wed, 31 Mar 2010 12:18:30 +0200 Subject: [M3devel] m3back directions? In-Reply-To: References: Message-ID: <20100331121830.22q33okrokkwsgwk@mail.elegosoft.com> Quoting Jay K : > I'm curious what, if anything, people are interested in in m3back? > > There are several mostly independent directions: > > - remove it; use the gcc backend or other (burg, llvm, generate C) > > - expand to support other targets, AMD64_*, including AMD64_NT > > m3objfile would need macho/elf support for non-NT It would be great if we could use the integrated backend for other targets, too. Adding ELF support will be a lot of work, but it's probably worth it. Olaf > - expand to generate good debugging information for Microsoft debuggers > > - various smaller/larger optimizations > > inlining? Inlining seems like the most lacking optimization and > thinking about it, it doesn't actually seem that hard to do, at > least a little bit. > > > > > > I'm thinking for now of working on the debugging information. > > > > > > - Jay > > > From jay.krell at cornell.edu Wed Mar 31 13:01:17 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 11:01:17 +0000 Subject: [M3devel] m3back directions? In-Reply-To: <20100331121830.22q33okrokkwsgwk@mail.elegosoft.com> References: , <20100331121830.22q33okrokkwsgwk@mail.elegosoft.com> Message-ID: > remove it? Oops, did I say that? :) Well, we should definitely do some of the others if are going to keep it. :) I'm leaning toward Microsoft CodeView debugging information currently. It should be the easiest and provide a lot of value. Though it is actually in m3objfile. I forgot to mention: It is already I think very portable to all 32bit x86 platforms. The real work work would be in m3objfile. A little in m3back regarding calling conventions perhaps. Maybe a little around munging function names for "-shared -fPIC" too. Like appending "@plt" in function calls or somesuch. - Jay > Date: Wed, 31 Mar 2010 12:18:30 +0200 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] m3back directions? > > Quoting Jay K : > > > I'm curious what, if anything, people are interested in in m3back? > > > > There are several mostly independent directions: > > > > - remove it; use the gcc backend or other (burg, llvm, generate C) > > > > - expand to support other targets, AMD64_*, including AMD64_NT > > > > m3objfile would need macho/elf support for non-NT > > It would be great if we could use the integrated backend for other > targets, too. Adding ELF support will be a lot of work, but it's probably > worth it. > > Olaf > > > - expand to generate good debugging information for Microsoft debuggers > > > > - various smaller/larger optimizations > > > > inlining? Inlining seems like the most lacking optimization and > > thinking about it, it doesn't actually seem that hard to do, at > > least a little bit. > > > > > > > > > > > > I'm thinking for now of working on the debugging information. > > > > > > > > > > > > - Jay > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Mar 31 13:15:08 2010 From: jay.krell at cornell.edu (Jay K) Date: Wed, 31 Mar 2010 11:15:08 +0000 Subject: [M3devel] changing m3middle/m3back to have multiple passes? Message-ID: I'm thinking m3back could use multiple passes. There a few reasons, things that would be easier/possible if it had them. I know adding passes will slow it down, but I suspect it will still be about as pleasantly fast as it is now. The optimizations I have in mind: inlining, including where function definitions come after the call I think inlining where the "small" functions come first is doable in pass, but not if uses come before calls. not saving unused registers This only a small optimization and could probably be done in one pass if m3objfile got a "memmove" operation. Well, I already have the optimization in somewhat, but it involves nop-ing out code instead of removing it completely. dead store removal, and then removing the resulting dead code to compute the value possibly better register allocation -- e.g. based on noticing which variables are used often, or based on how they are used, for example if a variable (or expression/temporary) ends serving a shift count, we should favor putting it in ecx up front, instead of moving it to ecx later, but we don't generally know this till too late As I understand, m3middle already has the notion of "recording" and "playing back" calls to the M3CG interface. That seems like a good starting point? Some optimizations can be made via "CG to CG" transforms. However I think in general I think the required design would include: - ability to add in "private operations"; maybe - ability to add "private data" to existing operations A very simple one is annotate functions with required frame size. This isn't currently known until the end of the function. - clearly, ability to remove operations Can/should we somehow augment m3middle in support of this line of thinking? Or is that just not the way to go? Each backend should have its own internal representation and loop over it? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 31 16:02:36 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 31 Mar 2010 10:02:36 -0400 Subject: [M3devel] m3back directions? In-Reply-To: References: Message-ID: <376BB9E6-AC9B-4110-AD22-83F8186BE5BF@cs.purdue.edu> On 31 Mar 2010, at 00:41, Jay K wrote: > I'm curious what, if anything, people are interested in in m3back? Simple. Fast. It is target-dependent so optimising there is misguided effort unless the opts are target-dependent (instruction selection, register allocation!). We don't have resources to maintain a sophisticated backend. Better to rely on other tools. There was a Linux version in pm3 that we should be able to model on. > There are several mostly independent directions: > - remove it; use the gcc backend or other (burg, llvm, generate C) > - expand to support other targets, AMD64_*, including AMD64_NT > m3objfile would need macho/elf support for non-NT > - expand to generate good debugging information for Microsoft debuggers > - various smaller/larger optimizations > inlining? Inlining seems like the most lacking optimization and thinking about it, it doesn't actually seem that hard to do, at least a little bit. > > > I'm thinking for now of working on the debugging information. > > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Mar 31 18:08:38 2010 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 31 Mar 2010 12:08:38 -0400 Subject: [M3devel] m3back directions? In-Reply-To: <20100331121830.22q33okrokkwsgwk@mail.elegosoft.com> References: <20100331121830.22q33okrokkwsgwk@mail.elegosoft.com> Message-ID: On 31 Mar 2010, at 06:18, wagner at elegosoft.com wrote: > Quoting Jay K : > >> I'm curious what, if anything, people are interested in in m3back? >> >> There are several mostly independent directions: >> >> - remove it; use the gcc backend or other (burg, llvm, generate C) >> >> - expand to support other targets, AMD64_*, including AMD64_NT >> >> m3objfile would need macho/elf support for non-NT > > It would be great if we could use the integrated backend for other > targets, too. Adding ELF support will be a lot of work, but it's probably > worth it. Have you looked at the pm3 Linux support? > > Olaf > >> - expand to generate good debugging information for Microsoft debuggers >> >> - various smaller/larger optimizations >> >> inlining? Inlining seems like the most lacking optimization and thinking about it, it doesn't actually seem that hard to do, at least a little bit. >> >> >> >> >> >> I'm thinking for now of working on the debugging information. >> >> >> >> >> >> - Jay >> >> >> > >